This PR makes the AI suggestions button more stable in case of
misconfigured fields:
* Only make the button clickable if there are sample values
* Filter out sample documents that don't have the required field on the
server (would have broken the request before)
<img width="344" alt="Screenshot 2025-03-04 at 15 43 23"
src="https://github.com/user-attachments/assets/12045985-cfac-4a13-a23c-595ac6503c1a"
/>
Embeddable ViewMode is part of legacy embeddable architecture. This PR
removes Embeddable ViewMode and replaces its usage with
presentation-publishing ViewMode. presentation-publishing ViewMode is a
string literal type so an enum is no longer needed.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Fixes [#212745](https://github.com/elastic/kibana/issues/212745)
## Summary
Fix for the hosts total count query, which was not correctly filtering
hosts from system integration and APM
<img width="584" alt="image"
src="https://github.com/user-attachments/assets/05c3b244-3a0f-425a-82db-3498c1a6dbb7"
/>
### How to test
- Connect Kibana to an edge cluster
- Navigate to Infrastructure > Hosts
- The total number of items on the table should be equal to the number
in the hosts count KPI
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This PR introduces a new script for loading parameterized workflow
insights into a data stream. It enables UI/UX testing without requiring
an agent installation or generating insights manually.
Arguments
```
--endpointId Required. The endpoint ID to use for generating workflow insights.
--elasticsearch Optional. The URL to Elasticsearch. Default: http://localhost:9200
--username Optional. The username to use for authentication. Default: elastic
--password Optional. The password to use for authentication. Default: changeme
--count Optional. The number of workflow insights to generate. Default: 5
--os Optional. The OS to use for generating workflow insights. Default: linux
--antivirus Optional. The antivirus to use for generating workflow insights. Default: ClamAV
--path Optional. The executable path of the AV to use for generating workflow insights. Default: /usr/bin/clamscan
```
Example usage:
* Load 5 workflow insights, using the default values - Linux, ClamAV,
/usr/bin/clamscan on the endpoint with ID
8ee2a3a4-ca2b-4884-ae20-8b17d31837b6
`node ./load_workflow_insights.js --endpointId
8ee2a3a4-ca2b-4884-ae20-8b17d31837b6`
* Load 10 workflow insights for Malwarebytes with path of C:\\Program
Files\\Malwarebytes\\Anti-Malware\\mbam.exe on Windows endpoint with ID
8ee2a3a4-ca2b-4884-ae20-8b17d31837b6
`node ./load_workflow_insights.js --endpointId
8ee2a3a4-ca2b-4884-ae20-8b17d31837b6 --count 10 --os windows --antivirus
Malwarebytes --path 'C:\\Program
Files\\Malwarebytes\\Anti-Malware\\mbam.exe'`
## Summary
This PR fixes the Unstructured syslog flow. It picks up 5 samples send
them to LLM to create a pattern and tests all the samples against the
pattern , collects the unparsed samples [ if any ] , send them in for
next round of pattern check and so on.
This creates a list of patterns that matches all the samples and creates
a grok processor with those patterns and it breaks the syslogs down into
a JSON for ECS mapping , categorization and related graphs.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Summary
Fix https://github.com/elastic/kibana/issues/211514
It seems the failure is because of the order difference in
`agent_logs_top_errors`.
The other differences shouldn't matter because we are using `
expect.objectContaining` on the result object.
Enabling `consistent-type-imports` rule and running `node scripts/eslint
--no-cache --fix` to fix every type import.
Every other ML owned plugin has this rule enabled. Originally added in
https://github.com/elastic/kibana/pull/176921
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
BUG: https://github.com/elastic/security-team/issues/11783
This PR fixes the behaviour of the
`/api/security_ai_assistant/chat/complete` route where the
`conversationId` can be passed as an empty string. This may lead to
unexpected results described in
https://github.com/elastic/security-team/issues/11783#issuecomment-2696529040.
### Expected behaviour
We should throw a bad request (400) http error when empty
`conversationId` has been passed.
### Testing
* Use this `curl` command to test the endpoint.
```
curl --location 'http://localhost:5601/api/security_ai_assistant/chat/complete' \
--header 'kbn-xsrf: true' \
--header 'Content-Type: application/json' \
--data '{
"connectorId": "{{my-gpt4o-ai}}",
"conversationId": "",
"isStream": false,
"messages": [
{
"content": "Follow up",
"role": "user"
}
],
"persist": true
}'
```
You should see next error as a response:
```
{
"statusCode": 400,
"error": "Bad Request",
"message": "[request body]: conversationId: String must contain at least 1 character(s), conversationId: No empty strings allowed"
}
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Fix for
[https://github.com/elastic/kibana/security/code-scanning/546](https://github.com/elastic/kibana/security/code-scanning/546)
To fix the problem, we need to ensure that backslashes are also escaped
in the `trim_key` and `trim_value` properties of the `kvInput` object.
This can be done by adding an additional replace call to escape
backslashes before escaping single and double quotes. The best way to
fix this without changing existing functionality is to use a regular
expression with the `g` flag to replace all occurrences of backslashes
with double backslashes.
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
## Summary
Refactors code in Asset Inventory page for simplicity and consistency.
### Changes
- Centralized constants for consistency
- Simplified `<AllAssets>` page, removed unused props, renamed
variables, etc...
- Encapsulated technical preview stuff into `<TechnicalPreviewBadge>`
- Removed deprecations in EUI components and styling
Also, this PR **deletes the mocked data** that was used before
integrating the UI with the backend.
### Questions
- Do we see value in centralizing all strings in a new file such as
`localized_strings.ts`?
### Out of scope
Hooks in `hooks/use_asset_inventory_data_table` and field selector
components were all duplicated from the CSP plugin. I haven't put effort
in refactoring those since we'll need to remove the duplication and make
them reusable
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [x] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Risks
No risk since code is still hidden behind the *Enable Asset Inventory*
advanced setting and the beta *Cloud Asset* integration must be
installed.
## Summary
The else is being called atm only in Discover ES|QL mode, when the user
is typing stats or keep.
The `isTextBased` flag is missing which means that we don't call the
`mapToOriginalColumnsTextBased` but the one for the dsl mode which might
cause bugs.
I don't think that Peter had any reason to do so, it seems more as it
slipped.
## Summary
Small follow up of https://github.com/elastic/kibana/pull/208091
The editor autocompletion added an incorrect import and so I'm removing
it, plus a few comments that should have been removed.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## 📓 Summary
When the condition is not met, the processing simulation reports wrong
metrics and fails on a unhandler error.
This work fix the issue and also update the document simulation metrics,
reporting how many documents are skipped by a processor during the
simulation.
A follow-up work will update the filters on the date to better reflect
the available states of the documents (parsed, partially parsed,
skipped, failed).
<img width="701" alt="Screenshot 2025-02-28 at 12 47 10"
src="https://github.com/user-attachments/assets/1b6979e4-78a1-4db3-af72-faaf06c0e249"
/>
## Summary
Resolves#206488. This PR introduces a new internal API that allows an
agent policy and its package policies to be created in one request:
<details>
<summary>⤵️ Expand for console snippet ⤵️</summary>
```
POST kbn:/internal/fleet/agent_and_package_policies
{
"id": "jens-awesome-policy",
"name": "awesome policy",
"description": "",
"namespace": "default",
"package_policies": [
{
"policy_ids": [
"jens-awesome-policy"
],
"package": {
"name": "log",
"version": "1.1.2"
},
"name": "log-for-awesome-policy",
"description": "",
"namespace": "",
"inputs": {
"logs-logfile": {
"enabled": true,
"streams": {
"log.log": {
"enabled": true,
"vars": {
"paths": [
"/tmp/some-path"
],
"data_stream.dataset": "generic",
"tags": [],
"custom": ""
}
}
}
}
}
},
{
"id": "fixed-id-for-filestream",
"package": {
"name": "filestream",
"version": "1.0.1"
},
"name": "filestream-1",
"description": "",
"namespace": "",
"inputs": {
"filestream-filestream": {
"enabled": true,
"streams": {
"filestream.generic": {
"enabled": true,
"vars": {
"paths": [
"/var/log/*.log"
],
"data_stream.dataset": "filestream.generic",
"parsers": "#- ndjson:\n# target: \"\"\n# message_key: msg\n#- multiline:\n# type: count\n# count_lines: 3\n",
"exclude_files": [
"\\.gz$"
],
"include_files": [],
"tags": [],
"recursive_glob": true,
"clean_inactive": -1,
"harvester_limit": 0,
"fingerprint": true,
"fingerprint_offset": 0,
"fingerprint_length": 1024,
"exclude_lines": [],
"include_lines": []
}
}
}
}
}
}
]
}
```
</details>
If successful, the response will be the agent policy with the full
nested package policies.
`id`s can be specified or omitted for either the agent or package
policy. If necessary, the resulting package policy's `policy_id` /
`policy_ids` will be updated with final ID of the created agent policy.
If any of the package policies fail to be created for any reason, there
is a rollback mechanism to delete the agent and package policies that
were already created.
The API also supports any query params that are supported by the create
agent policy and create package policy endpoints:
```
sys_monitoring: boolean; // passed to agent policy creation
format: 'simplified' | 'legacy; // passed to package policy creation
```
### Dev note
The new `createAgentAndPackagePoliciesHandler()` is unique in that it
acts mostly as a passthrough to other handlers,
`createAgentPolicyHandler` and `createPackagePolicyHandler`. This means
all the checks (spaces, rbac, etc) performed on the other handlers are
triggered appropriately.
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
<img width="600" alt="Screenshot 2025-02-25 at 11 35 32 AM"
src="https://github.com/user-attachments/assets/2567e20c-1223-4001-9866-edbe8c534350"
/>
PR does the following
* Updates maplibre to 5.1.1. Maplibre 5.0 introduced the globe
projection
* Updates sample data maps to use new globe projection
* Existing maps will use existing "mercator" projection
* Adds globe projection to newly created maps
* Adds `projection` setting, allowing users to switch between classic
"mercator" and new "globeInterpolate" projection
<img width="400" alt="Screenshot 2025-02-25 at 11 35 41 AM"
src="https://github.com/user-attachments/assets/5c55003b-dbe9-4adc-835c-0a2911c81074"
/>
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Fixes logic on fresh cluster where the ELSER was not started yet, in
this case API reports `status` as `uninstalled`, but it doesn't mean
that the Product documentation was actually uninstall, but rather it's a
default state.
Added internal `product_documentation_status` to KB status API to make
sure we keep track of the status internally and present the banner only
if the docs were intentionally uninstalled
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR exposes `windows.ransomware.dump_process` as an advanced policy
option for Elastic Defend.
If enabled, this option will make the endpoint generate a memory dump of
the ransomware process before killing it, assisting the ransomware
investigation process.
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Part 3 of https://github.com/elastic/kibana/pull/212173
### Testing
For setup see testing section here:
https://github.com/elastic/kibana/pull/212173#issue-2870522020
Testing this work is a bit more `console.log` related, as the primary
aim of many of these changes was to limit the amount of re-renders
caused by state changes within many of the sourcerer utils. React is
relatively fast at handling many of these re-renders, but considering
the widespread usage of `useSourcererDataView` and the top level
location of `useInitSourcerer`, it was worth taking care of some low
hanging fruit.
### Background
The changes in this PR centered around limiting the number of re-renders
that were being caused unnecessarily in `useSourcererDataView` and
`useInitSourcerer` by identifying some quick wins.
Comments regarding the changes can be found in the PR diffs
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### Identify risks
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
**Resolves: https://github.com/elastic/kibana/issues/208361**
**Resolves: https://github.com/elastic/kibana/issues/210544**
## Summary
This PR introduces significant memory consumption improvements to the
prebuilt rule endpoints, ensuring users won't encounter OOM errors on
memory-limited Kibana instances.
Memory consumption testing results provided in
https://github.com/elastic/kibana/pull/211045#issuecomment-2689854328.
## Details
This PR implements a number of memory usage optimizations to the
prebuilt rule endpoints with the final goal reducing chances of getting
OOM errors. The changes are extensive and require thorough testing
before merging.
The changes are described by the following bullets
- The most significant change is the addition of pagination to the
`upgrade/_review` endpoint. This endpoint was known for causing OOM
errors due to its large and ever-growing response size. With pagination,
it now returns upgrade information for no more than 20-100 rules at a
time, significantly reducing its memory footprint.
- New backend methods, such as
`ruleObjectsClient.fetchInstalledRuleVersions`, have been introduced.
These methods return rule IDs with their corresponding installed
versions, allowing to build a map of outdated rules without loading all
available rules into memory. Previously, all installed rules, along with
their base and target versions, were fetched unconditionally before
filtering for updates.
- The `stats` data structure of the review endpoint has been deprecated
(it can be safely removed after one Serverless release cycle). Since the
endpoint now returns paginated results, building stats is no longer
feasible due to the limited rule set size fetched on the server side. As
the side effect it required removing related Cypress tests asserting
`Update All` disabled when rules can't be updated.
- All changes to the endpoints are backward-compatible. All previously
required returned structures still present in response. All newly added
structures are optional.
- Upgradeable rule tags are now returned from the prebuilt rule status
endpoint.
- The frontend logic has been updated to move sorting and filtering of
prebuilt rules from the client side to the server side.
- The `upgrade/_perform` endpoint has been rewritten to use lightweight
rule version information rather than full rules to determine upgradeable
rules. Additionally, upgrades are now performed in batches of up to 100
rules, further reducing memory usage.
- A dry run option has been added to the upgrade perform endpoint. This
is needed for the "Update all" rules scenario to determine if any rules
contain conflicts and display a confirmation modal to the user.
- An option to skip conflicting rules has been added to the upgrade
endpoint when called with the `ALL_RULES` mode.
- The `install/_review` endpoint's memory consumption has been optimized
by avoiding loading all rules into memory to determine available rules
for installation. Redundant fetching of all base versions has also been
removed, as they do not participate in the calculation.
---------
Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co>
## Summary
Relates https://github.com/elastic/kibana/issues/206242
Implemented installing integrations from the doc in the follower index.
Can be tested locally by creating the ccr index locally or setting up 2
local clusters and set up ccr between them.
To test:
- Requires https://github.com/elastic/elasticsearch/pull/123493,
checkout and run es from source
```
yarn es source --license trial -E xpack.security.authc.api_key.enabled=true -E xpack.security.authc.token.enabled=true --source-path=/Users/juliabardi/elasticsearch -E path.data=/tmp/es-data -E xpack.ml.enabled=false
```
- Enable feature flag `xpack.fleet.enableExperimental:
['enableSyncIntegrationsOnRemote']`
- Create doc in ccr index, `hosts` should match local elasticsearch host
```
POST fleet-synced-integrations-ccr-remote1/_doc
{
"id": "fleet-synced-integrations",
"remote_es_hosts": [
{
"hosts": [
"http://192.168.64.1:9200"
],
"name": "remote1",
"sync_integrations": true
}
],
"integrations": [
{
"package_version": "1.25.0",
"package_name": "nginx",
"updated_at": "2025-02-24T09:03:51.936Z"
}
]
}
```
- Wait 1m until the task runs, verify that the integrations from the doc
are installed
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
To update the `global_telemetry_config` flag in Defend package policies,
we subscribe to the Telemetry plugin's `isOptedIn$` observable during
Kibana's `start()` phase, and receive the initial value immediately.
This feature is used for 'migrating' existing package policies: after
stack upgrade, when Kibana starts up, this subscription mechanism makes
sure that existing policies are backfilled with the new field.
But not on cloud and serverless instances.
It turned out, that while this works on local instances, on cloud and
serverless instances, at the very moment we receive the value during
`start()`, some mechanisms are not yet green, and this resulted in
`security_exception: missing authentication credentials for REST
request` when trying to read Saved Objects.
As subscribing to`core.status.core$`, and waiting until `ServiceStatus`
for `elasticsearch` and `savedObjects` is `available` didn't solve the
issue, I simply added a retry mechanism, which, at least, protects
against other temporary issues as well.
Some additional logging is added as well.
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
