Closes https://github.com/elastic/observability-dev/issues/4563
## Summary
This PR updates the Obs AI Assistant system prompt to explicitly inform
the LLM about the presence of anonymized entities (e.g., hashes or
placeholder tokens). The goal is to prevent the LLM from attempting to
interpret or hallucinate the meaning of these anonymized tokens.
What Changed
- Modified the system prompt to include a new instruction:
```txt
Some entities in this conversation (like names, locations, or IDs) have been anonymized using placeholder hashes (e.g., `PER_123`, `LOC_abcd1234`). These tokens should be treated as distinct but semantically unknown entities. Do not try to infer their meaning. Refer to them as-is unless explicitly provided with a description.
```
- This instruction is now included in all prompts sent to the LLM as
part of the chat completion setup when there are anonymization rules.
# Overview
This pull request adds capabilities associated with the [Privileged
Access Detection (PAD) anomaly integration
package](https://www.elastic.co/docs/reference/integrations/pad) as a
first-class-citizen into the Entity Analytics Privileged User Monitoring
feature.
# How to test
- Pull this branch into your local machine
- Ensure that the security experimental flag
`privilegeMonitoringEnabled` has been enabled, for example by setting
`xpack.securitySolution.enableExperimental:
[privilegeMonitoringEnabled]` in your `config/kibana.dev.yml` file
- Start Elasticsearch and Kibana
- From the
[security-documents-generator](https://github.com/elastic/security-documents-generator)
repository, run the following command: `yarn start
privileged-user-monitoring` (ensuring your config is pointing to your
locally running Elastic cluster). This will load "source" events that
are anomalous in nature. You can run this command more than once if
desired to upload more than 10 users.
- Grab the CSV file that the generator created in its console output.
- Open the Entity analytics page and navigate to dashboards (by clicking
"Go to dashboards")
- Add the privileged users from the previous step to the privileged
users index using the CSV File Upload option.
> [!NOTE]
> Any errors regarding risk scoring are unrelated to this PR, and are
being resolved separately
- You should see a panel that says "Enable Privileged access detection".
Click "Install", and you'll meet a loading state.
- Once complete, you shouldn't see any results. That's because, even
though we **install** the ML jobs by default, we don't **run** them by
default.
- Click "ML Job Settings", and note that only `pad` jobs should be
displaying in this callout. Feel free to test this callout's links and
filtering options.
- Click "Run job" next to the job called
`pad_linux_high_count_privileged_process_events_by_user`. This is the
job for which we have anomaly data.
- Click away from the callout. **You still shouldn't see data.** That's
because there just aren't any anomalies "today".
- Change the global date filter at the top of the screen to "Last 30
days".
- You should see something similar to this:
<img width="1441" alt="Screenshot 2025-06-16 at 12 50 25 AM"
src="https://github.com/user-attachments/assets/2b3f11f2-f45d-4716-bb8e-79d2b585aa3e"
/>
- Congrats for making it this far! Some things to play around with:
- Click around on the anomaly filters (i.e., click 25-50), and notice
that the results will change.
- Ensure that the ordering of the users is based on the **highest single
anomaly score in any visible bucket**. Meaning, if user `samwise` has
only one anomaly, but its score is 99, and user `frodo` has dozens of
anomalies, but no higher than, say, 80, `samwise` will be on the top of
the list.
- Click the user names to open the appropriate user flyout
- Change the global time filter to change ranges, and ensure the data
shows up appropriately. The buckets will try to roughly show 30 buckets
total, but will have a maximum of 3 hours, meaning if your range is too
small, it may show fewer than 30 buckets. This is intentional behavior,
as the PAD jobs have an anomaly job window of 3 hours.
- Click the button that says "View all in Anomaly Explorer", which uses
the currently selected global time range, and compare results
> [!WARNING]
> Remember that the users in the privileged user monitoring table are
only those that you designated as privileged users in a previous step.
In contrast, the Anomaly Explorer page shows **all** users.
Additionally, note that there may be very slight differences between the
swimlanes, because ES|QL calculates the bucket dates slightly
differently than the Anomaly Explorer. **This should not affect the
results themselves, only the buckets that an individual anomaly score
might fall in.** You might see a single anomaly fall into one visual
bucket instead of another, but the date should be correct.
# Helpful hints
If you'd like an easy way to "reset" the Privileged Access Detection
package and delete its associated jobs (and anomaly data), so that you
can redo the onboarding flow, you can run the below commands (changing
any credentials as necessary):
```shell
curl "http://localhost:5601/api/fleet/epm/packages/pad/0.5.0" \
-X 'DELETE' \
-H 'elastic-api-version: 2023-10-31' \
-H 'kbn-xsrf:true' \
--user elastic:changeme
curl "http://localhost:5601/internal/ml/jobs/delete_jobs" \
-H 'kbn-xsrf:true' \
--user elastic:changeme \
-X 'POST' \
-H 'elastic-api-version: 1' \
-H 'x-elastic-internal-origin:kibana' \
-H 'Content-Type: application/json' \
--data-raw '{
"jobIds": [
"pad_linux_high_count_privileged_process_events_by_user",
"pad_linux_high_median_process_command_line_entropy_by_user",
"pad_linux_rare_process_executed_by_user",
"pad_okta_high_sum_concurrent_sessions_by_user",
"pad_okta_rare_host_name_by_user",
"pad_okta_rare_region_name_by_user",
"pad_okta_rare_source_ip_by_user",
"pad_okta_spike_in_group_application_assignment_changes",
"pad_okta_spike_in_group_lifecycle_changes",
"pad_okta_spike_in_group_membership_changes",
"pad_okta_spike_in_group_privilege_changes",
"pad_okta_spike_in_user_lifecycle_management_changes",
"pad_windows_high_count_group_management_events",
"pad_windows_high_count_special_logon_events",
"pad_windows_high_count_special_privilege_use_events",
"pad_windows_high_count_user_account_management_events",
"pad_windows_rare_device_by_user",
"pad_windows_rare_group_name_by_user",
"pad_windows_rare_privilege_assigned_to_user",
"pad_windows_rare_region_name_by_user",
"pad_windows_rare_source_ip_by_user"
],
"deleteUserAnnotations": true,
"deleteAlertingRules": false
}
'
```
# What's left?
- This PR does not implement "proactive" permissions checks. Instead,
relevant permission issues will cause requests/queries to fail, which
are appropriately shown in the UI for troubleshooting. This proactive
check will be tackled in [this (private) followup
issue](https://github.com/elastic/security-team/issues/12822).
- There is not yet a way to "upgrade" the package directly from the
Privileged User Monitoring screen. This would be a nice addition later
on, instead of requiring users to navigate to the integration page. This
behavior will be tackled in [this (private) followup
issue](https://github.com/elastic/security-team/issues/12823).
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
This PR updates Asset Inventory Grouping and Datagrid functionalities to
accommodate the `asset.criticality` soft delete mechanisms, which sets
the value of `asset.criticality` to "delete" once an entity is manually
"Unassigned" after having a value before:
- Added condition on grouping to treat both "deleted" and "missing
value" as Unassigned.
- Added rendering condition on the datatable to display the Unassigned
badge when `asset.criticality` value is either missing or "deleted".
### Recording
https://github.com/user-attachments/assets/37d7e44f-cb57-4c29-b49d-cda9b341497d
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This PR contains the following updates:
| Package | Type | Update | Change |
|---|---|---|---|
| [form-data](https://redirect.github.com/form-data/form-data) |
devDependencies | patch | [`^4.0.2` ->
`^4.0.3`](https://renovatebot.com/diffs/npm/form-data/4.0.2/4.0.3) |
---
### Release Notes
<details>
<summary>form-data/form-data (form-data)</summary>
###
[`v4.0.3`](https://redirect.github.com/form-data/form-data/compare/v4.0.2...v4.0.3)
[Compare
Source](https://redirect.github.com/form-data/form-data/compare/v4.0.2...v4.0.3)
</details>
---
### Configuration
📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR has been generated by [Renovate
Bot](https://redirect.github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4xMDcuMCIsInVwZGF0ZWRJblZlciI6IjM5LjEwNy4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJUZWFtOm9icy11eC1tYW5hZ2VtZW50IiwiYmFja3BvcnQ6YWxsLW9wZW4iLCJjaTphbGwtY3lwcmVzcy1zdWl0ZXMiLCJyZWxlYXNlX25vdGU6c2tpcCJdfQ==-->
---------
Co-authored-by: elastic-renovate-prod[bot] <174716857+elastic-renovate-prod[bot]@users.noreply.github.com>
Co-authored-by: Jon <jon@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
# Overview
This PR does various small things regarding the Privileged user
monitoring and Entity Analytics workflow, including:
- Adds technical preview labels where appropriate
- Adds in the screenshots for the navigation
- Removes the onboarding video in favor of an onboarding illustration
- Temporarily removes the integration cards from onboarding
# How to test
- Pull the code down and ensure the `privilegeMonitoringEnabled`
experimental flag is set
- Go to the "Entity Analytics" navigation menu in classic nav, and see
the new screenshots
- See that the Privileged user monitoring onboarding page should show
"Technical Preview"
- See that the Privileged user monitoring onboarding page should not
show Integration cards
- See that the video in the onboarding has been replaced with an
illustration
- Click "Go to dashboards =>", and see that the Privileged user
monitoring onboarding page should show "Technical Preview"
## 🧵 Summary
This PR introduces a few structural and cleanup improvements:
1. Moves core Emotion CSS helpers (`useMemoizedStyles` renamed as
`useMemoCss`, `kbnFullBodyHeightCss`, `kbnFullScreenBgCss`) from
`core/public` to a new package: `@kbn/css-utils`.
2. Removes a significant portion of legacy SCSS from the core plugin.
3. Replaces scss mixin with emotion `kbnFullScreenBgCss` across Kibana
(we have scss and emotion version, but emotion version wasn't widely
used yet).
4. As a result of (3), some plugin tests were migrated to React Testing
Library. This was necessary because Emotion-generated snapshots in
Enzyme were difficult to read and maintain when moving to emotion.
### Considerations
I initially tried to add the package to the [shared-deps
bundle](8e15517ddd),
but couldn’t get the SVG imports for `kbnFullScreenBgCss` to work
correctly in that setup.
As a workaround, I opted to import the helpers directly from their
source files.
An alternative approach could be to convert the used SVGs into React
components and use those within the shared package. Or explore something
like a static package and try to somehow wire in that in the webpack
internal compilers, but it doesn't seem to be worth the effort at the
moment.
### 💡 Motivation
- These utils don’t need to live in Core and are now decoupled to
improve performance and flexibility.
- Importing from `core/public` (even just for a small hook) was adding
noticeable overhead to test runs:
- ~1–2s delay on first Jest execution
- ~200ms added on subsequent runs
- Occasional CI timeouts due to deep import graph
### 👥 Ownership
I assigned this package to sharedux team. Thank you!
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Addresses https://github.com/elastic/kibana/issues/151636 .
This test was leftover from migration. Does not appear to be used.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Adding license check for list and disable scheduled report APIs
## To Verify
1. Run ES with basic license
2. Try accessing the scheduled report list and disable APIs from the Dev
Console. You should get a 403 error.
This PR removes the current GET user route as it works with IDs and we
do not use IDs in privmon.
We'd rather remove this before it gets documented and pushed out. We can
revert this PR later if needed
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Mark Hopkin <mark.hopkin@elastic.co>
## Summary
The following changes are in support of space awareness:
- A new reference data item type to store the space ID that should be
used to display orphaned response actions
- With space awareness, where the visibility of response actions is tied
to the Fleet integration policy associated with the agents, its possible
that response actions (older one) may become unaccessible when the
associated integration package policy is deleted. To mitigate this for
customer that may want to continue to see them, we will allow for a
specific space ID to be defined where these should be made visible
- Two new **internal** APIs to read and update the space id where orphan
response actions should be visible
- Both APIs are currently only allowed to be used by any user having the
`superuser` role
- Read API: `GET /internal/api/endpoint/action/_orphan_actions_space`
- Update API: `POST /internal/api/endpoint/action/_orphan_actions_space`
- with a `body` of `{ spaceId: "space_id_here" }`
- `spaceId` can also be set to an empty string (default out of the box),
which results in the orphan response actions not being display in any
space
- A new internal `ReferenceDataClient` for managing reference data
items.
This makes the existing 'deactivate_all_instrumentations' and
'deactivate_instrumentations' agent configuration settings
available to EDOT Node.js agents (`agent.name:
opentelemetry/nodejs/elastic`).
**Addresses:** https://github.com/elastic/kibana/issues/202068
## Summary
This PR actualizes prebuilt rule customization test plan with the current feature state and desired test coverage we expect for prebuilt rule customization sub-feature.
## Details
The changes include
- Adding an extra test scenario for prebuilt rule customization via public API
- Fixing a scenario under Licensing section
- Fixing wording to make scenarios simpler
Closes https://github.com/elastic/kibana/issues/223954
Closes https://github.com/elastic/kibana/issues/223921
## Summary
This PR adds support for ESQL autocompletion inside single quotes, which
allows suggestions that include triple quotes such as `KQL(``````)`.
<img width="1165" alt="Screenshot 2025-06-19 at 13 39 51"
src="https://github.com/user-attachments/assets/8ff59bf7-534e-4539-8cf8-2a1145dbf842"
/>
**Changes include:**
- The lexer rules have been updated to recognize single-quote query as
ESQL so that the highlighting is correct
- We escape and unescape characters that are invalid in a Console string
when we suggest inside single quotes so that we don't cause errors in
Console
**How to test:**
Make sure that ESQL suggestions are displayed inside both `"query":
"...` and `"query": """...` and that suggestions that contain quotes are
correctly escaped inside single-quote queries.
---------
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
## Summary
Summarize your PR. If it involves visual changes include a screenshot or
gif.
On cloud connector, cloud formation redirection kibana replaces the
value of the parameter param_ElasticResourceId=RESOURCE_ID with:
We used the deployment ID for hosted. While in ESS, we need to replace
with Kibana Component ID for hosted, instead of Deployment ID. We
extract `cloud.cloudId` base64 and decode `cloudId` in the following
format
`deployment_name:base64(<host>$<es_cluster_id>$<kibana_component_id>)`
and get the Kibana Component ID
Resolves: #213391
This PR adds validation for task interval where it is updated.
There are 5 places in the TaskStore that we update the interval:
`schedule`, `bulkSchedule`, `update`, `bulkUpdate` and
`bulkPartialUpdate`
For the bulk operations we just skip the task with invalid interval, for
schedule and update throw an error and stop the execution.
## To verify:
Create a rule with connector.
Let it run for a while.
Then update the rule.
There shouldn't be any error.
**Addresses:** https://github.com/elastic/kibana/issues/223446
## Summary
This PR fixes an issue when time duration normalized to day(s) is shown as 0 seconds. The fix is performed by allowing using days time unit at rule schedule.
## Details
The issue happens when rule schedule's look-back gets normalized to day(s). The reason is that look-backs input doesn't support Days time unit. It leads to inability to parse the value and displaying the default value which is 0 seconds.
Rule schedule is shown to the users as rule `interval` and `look-back` while rule's SO saves the schedule by using three fields `interval`, `from` and `to`. Where `look-back` represents a logical value calculated as `lookback` = `to` - `from` - `interval`. Taking that into account it's becomes harder to maintain the original time duration unit value during prebuilt rules upgrade workflow (See https://github.com/elastic/kibana/pull/204317 for more details).
The easiest way to fix this issue is to allow Days time unit in rule schedule inputs. On top of that 24 hours are always 1 day making hours the largest simply convertible time unit. The PR allows hours in rule schedule.
**Before:**
https://github.com/user-attachments/assets/4f2038f1-4a6a-4a88-b86e-381a5b717605
**After:**
https://github.com/user-attachments/assets/74875bf2-9341-425f-a35f-c8b088c1ef6a
## Summary
* Address https://github.com/elastic/kibana/issues/217145 - Put in place
a check to ensure we're upgrading from Kibana 8.18.0 or newer.
* Address https://github.com/elastic/kibana/issues/220521 - New attempt
at removing the `switchToModelVersionAt` property, inspired on
https://github.com/elastic/kibana/pull/219029.
The previous attempt caused a regression: index meta information started
storing _modelVersions_ that were older than the previously stored ones,
which were defaulting to 10.0.0 for SO types that did NOT define
`modelVersions`.
This was due to the removal of the applyTypeDefaults, which was ensuring
all SOs had the `switchToModelVersionAt` property set.
This flag was then used by
`src/core/packages/saved-objects/base-server-internal/src/model_version/version_map.ts`
to determine whether to use `modelVersions` or the legacy `migrations`
property in order to determine the latest model version for a given
type.
When removing the `switchToModelVersionAt` flag (and its default
backfill), the logic started defaulting to the latest `migrations`
version for those SO types that were not defining any `modelVersion`,
resulting in older versions that those stored in the SO indices. This
caused incident https://elasticco.atlassian.net/browse/INC-3818.
This regression has been shipped in 9.0.0 (the PR was
[backported](https://github.com/elastic/kibana/pull/219329)), so in top
of the cleanup, we now need to address
https://github.com/elastic/kibana/issues/220521 to ensure a smooth
transition _OnPrem => Serverless_.
---------
Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
In Universal Profiling views, avoid to display the 0 value address
appended to executable file names.
Planned improvements on aggregations in the ES profiling plugin require
to reset the address for unsymbolized or partly symbolized frames.
For example, aggregating Kernel frames from two different hosts with
different kernel versions will currently end up with two distinct
entries in the TopN functions:
```
vmlinux: _raw_spin_unlock_irqrestore
vmlinux+0x124
vmlinux: _raw_spin_unlock_irqrestore
vmlinux+0x542
```
When aggregating these into a single line, displaying the address would
be misleading or wrong.
So the ES plugin may set the address to 0 for the aggregated entry
```
vmlinux: _raw_spin_unlock_irqrestore
vmlinux+0x0
```
Now, the `+0x0` seems superfluous and is not helpful to the user.
This PR would drop the address.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Related to https://github.com/elastic/kibana/issues/221521
This PR updates the logic that determines whether a clicked node in the
waterfall is a span or a transaction, making it compatible with
non-processed OTel traces data.
The approach used is the same as in the document profile enablement:
d80885a454/src/platform/plugins/shared/discover/public/context_awareness/profile_providers/observability/traces_document_profile/span_document_profile/profile.ts (L70-L79)
**Extra fix**
- Added a focus trap to the flyout because without it, focus sometimes
stayed on the waterfall behind, causing unexpected behavior when trying
to filter in the "Table" tab.
## How to test
- Enable the discover profiles by adding this to the` kibana.yml `file:
```discover.experimental.enabledProfiles:
- observability-traces-data-source-profile
- observability-traces-transaction-document-profile
- observability-traces-span-document-profile
```
- Make sure your space has Observability as Solution View.
- Open Discover and query non-processed OTel traces
- Open any flyout and go to the full screen waterfall
- Click on any node of the trace, it should be opened as "Span document"
## Summary
Closes https://github.com/elastic/kibana/issues/224596
The “View related errors” link in the full-screen waterfall was
resolving to an invalid ES|QL query in some specific scenarios (it
worked fine in edge-oblt, for example).
To work around this and make the query less restrictive for ES|QL, we’ve
updated it to use `QSTR` instead. It provides the same type of search
and gives us the expected results without triggering errors.
**Before**
**After**
## How to test
- Enable the discover profiles by adding this to the` kibana.yml `file:
```discover.experimental.enabledProfiles:
- observability-traces-data-source-profile
- observability-traces-transaction-document-profile
- observability-traces-span-document-profile
```
- Make sure your space has Observability as Solution View.
- Open Discover and query a trace with errors
- Open any flyout and go to the full screen waterfall
- Click on "View related links" on any of the nodes
Fixes https://github.com/elastic/kibana/issues/204668
## Summary
This PR adds a custom context menu to Console that correctly copies,
cuts, and pastes the selected text in the editor. Previously, when we
did right click (or Ctrl + click), the native browser context menu is
open but only the word on which we clicked was selected, rather than the
whole selected text in the editor. This is because the native browser
context menu is not aware of Monaco's selection and treats the text in
the editor as regular text in the browser. To fix this, we add a custom
context menu that is aware of Monaco editor's selection, which correctly
copies and cuts the selected text.
**Note for changes in CodeEditor component**: In order to enable the
custom context menu, we need to add the `import
'monaco-editor/esm/vs/editor/contrib/contextmenu/browser/contextmenu.js';`
import, which would enable the Monaco context menu in all code editors
in Kibana unless we explicitly disable it with `contextmenu: false`,
which we do in the CodeEditor component as a default behavior. This
ensures that all other code editors in Kibana still have the native
browser context menu. @elastic/appex-sharedux please let us know if you
would like us to add this custom context menu to the Code editor
component so that all editors in Kibana have the same context menu with
a correct behaviour.
https://github.com/user-attachments/assets/ceb65438-a17e-46fd-be6a-a1cce4d98df7
This PR contains the following updates:
| Package | Update | Change |
|---|---|---|
| docker.elastic.co/wolfi/chainguard-base-fips | digest | `799fc6d` ->
`d55453a` |
---
### Configuration
📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR has been generated by [Renovate
Bot](https://redirect.github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4xMDcuMCIsInVwZGF0ZWRJblZlciI6IjM5LjEwNy4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJUZWFtOk9wZXJhdGlvbnMiLCJiYWNrcG9ydDpza2lwIiwiY2k6YnVpbGQtZG9ja2VyLWZpcHMiLCJyZWxlYXNlX25vdGU6c2tpcCJdfQ==-->
Co-authored-by: elastic-renovate-prod[bot] <174716857+elastic-renovate-prod[bot]@users.noreply.github.com>
## Summary
This change is in support of space awareness which is currently behind a
feature flag:
- Adds new processing logic to the Fleet package policy delete extension
point to update response action that reference that package policy with
a tag indicating that policy was deleted
- This check is done for all packages that support response actions -
currently `endpoint`, `crowdstrike`, `microsoft_defender_endpoint` and
`sentinel_one`
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
The following changes are being done in support of space awareness:
- Ensure that a `tags` field is added to the response actions request DS
index
- FYI: the elastic defend package will soon also be updated with this
new field
- Update migration of response actions to populate the new `tags` field
with `INTEGRATION-POLICY-DELETED` for cases where the migration is
unable to determine the policy information associated with the agent(s)
the action was sent to
- Update internal response action utilities that retrieve action request
to ensure `tags` is an array
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
This PR updates the ES|QL grammars (lexer and parser) to match the
latest version in Elasticsearch.
Changes:
- moves sample on tech preview
<img width="818" alt="image"
src="https://github.com/user-attachments/assets/24441300-72c5-4055-b316-6e9ad3939d0a"
/>
- Sample now supports variables
- Fork is on tech preview but I didn't do this at this PR, I will do on
a follow up PR
- there is a breaking change in from, we dont allow anymore
"index"::"selector", fixed the tests and the parsers
- small changes in like, rlike expressions
---------
Co-authored-by: Stratoula <efstratia.kalafateli@elastic.co>
## Summary
We have [this RFC](https://github.com/elastic/ecs/pull/2493) in, I think
this is a safer bet and might save us a migration in the future:
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fix https://github.com/elastic/search-team/issues/10121
Add a base set of retrieval oriented tools, and expose them to the
default agent.
## Tools
**Warning: names are still TBD**
We're starting to see two different "layers" of tool appear: "simple"
tools, which are doing a simple programmatic (understand: no LLM) task,
and "smart" tools, which are more like workflows, with some of the steps
relying on a LLM.
This PR introduces the following tools:
### Simple tools
Simple tools (name TBD) are "programmatic" tools not relying on an LLM
for their execution.
This PR introduces this base set of tools:
- `get_document_by_id`: resolve a full document based on its id/index.
- `list_indices`: list the indices the current user has access to.
- `get_index_mappings`: retrieve the full mappings based for a given
index.
- `execute_esql`: executes a provided ES|QL query
### Smart tools
Smart tools can have multiple internal steps (even if it remains an
implementation detail), and are using LLM calls for some, or all, of
them.
*Note: there are huge potential areas of improvement in the current
implementation of all those smart tools. One of the intent of this work
is precisely to identify such areas of improvement*
#### `index_explorer`
Based on a natural language query, returns a list of indices that should
be searched, and their corresponding mappings.
<img width="984" alt="Screenshot 2025-06-17 at 16 44 32"
src="https://github.com/user-attachments/assets/edff3964-31e3-40ea-a761-adf7c45fcb17"
/>
#### `generate_esql`
Based on a natural language query, generates an ES|QL query.
- use the `nl-2-esql` task under the hood
- optional use `index-explorer` if `index` is not specified.
<img width="891" alt="Screenshot 2025-06-17 at 16 51 56"
src="https://github.com/user-attachments/assets/ce141d6b-dd4f-4eb9-ab32-823b81bc810b"
/>
#### `relevance_search`
Perform a "full-text search" based on given term and returns the most
relevant highlights.
<img width="1071" alt="Screenshot 2025-06-17 at 16 59 49"
src="https://github.com/user-attachments/assets/1f873e70-e277-424d-93e4-24b269a554e5"
/>
#### `natural_language_search`
Retrieve data based on a natural language query.
Converts a natural language query to an ES|QL one then executes it,
useing `generate_esql` and `execute_esql` under the hood.
<img width="768" alt="Screenshot 2025-06-18 at 08 31 52"
src="https://github.com/user-attachments/assets/cb319831-17ed-4ad7-9e1f-2fe90c2472fa"
/>
## Researcher assistant
The second part of this PR is implementing a researcher agent for deep
research tasks.
The researcher assistant is following a very classic
"act->process->reflect" cycle.
<img width="960" alt="Screenshot 2025-06-18 at 09 16 17"
src="https://github.com/user-attachments/assets/c24be323-ecf2-4c43-88fb-eaf874b18afc"
/>
The implementation of the cycle is currently as follow:
**1. Act**
Given a research topic, the research history and a list of tools, select
the tool best suited to search for this topic, and call it.
The tools exposed to the agent in this phase are:
- `index_explorer`
- `relevance_search`
- `nl_search`
*Note: later the whole `act` step could evolve to instead call sub
search agent with planning and multi-step execution.*
**2. Process**
Process the results from the latest `act` phase and create a
corresponding entry in the search log.
At the moment, we're simply storing the whole tool call + results to the
search log in a LLM-friendly format.
**3. Reflect**
Based on the main research query and the search log, identify where the
information collected are enough to answer the question. If not,
identify follow-up questions or sub-problems that it would be useful to
solve to gather more information
## What is out of scope of the current PR
- Figuring out which set of tools should be exposed by default to the
main agent (right now, all the tools listed in this PR are)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
The PR skips test suites for ES `9.0.*` which were failing when these
tests ran in Kibana `8.19` branch against ES `9.0.*` in forward
compatibility runs. The tests were failing because the index privilege `read_failure_store` isn't available available in ES 9.0.
## Summary
This PR introduces the concept of internally managed rule types. The
purpose of this PR is to hide alerts in the alerts table in the UI
produced by internally managed rule types. In following PRs, we will
enhance the framework to handle more cases when the product requirements
are clearer. If, in the future, the streams team wants to use the alerts
table to show stream alerts, we could introduce a new parameter in the
alerting API to allow alerts produced by internally managed rule types
to be returned.
Fixes: https://github.com/elastic/kibana/issues/221379
cc @kdelemme @dgieselaar
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
