## Summary
Fix and unskip TSVB Functional test by using the right Borealis colors.
fix https://github.com/elastic/kibana/issues/206380
---------
Co-authored-by: nickofthyme <nicholas.partridge@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This PR updates the method for fetching file events used as the
foundation for creating insights. Previously, we retrieved the last 200
events from the past 24 hours. With these changes, we now rely on
aggregations: all file events from the past 24 hours are aggregated by
file path, and for each path, only the latest event associated with it
is selected. The limit of 200 paths remains unchanged.
Resolves https://github.com/elastic/eui-private/issues/171
Resolves https://github.com/elastic/eui-private/issues/177
## Summary
This PR addresses a prior PR review
[comment](https://github.com/elastic/kibana/pull/203840/files#diff-bb850523655bac7adb30995553acabae9705435fa51e5b8bf13c483152db694a)
by removing `isServerless` from the logic determining what theme should
be used at runtime with a simple YML configuration setting instead.
I added a non-public `uiSettings.experimental.defaultTheme` config
property that defaults to `borealis` and is set to `amsterdam` in
`serverless.yml`. Since the default theme is now (and should be) set to
Borealis, I also updated `DEFAULT_THEME_NAME` and `FALLBACK_THEME_NAME`
to reflect that. This doesn't have any impact on Serverless; it will
keep using Amsterdam.
Additionally, while making these changes, I wanted to simultaneously
improve types and address earlier PR
[comment](https://github.com/elastic/kibana/pull/199748#discussion_r1840402343).
Now `SUPPORTED_THEME_NAMES` array is declared as `const` making the
`ThemeName` type strict instead of resolving a generic `string` type.
Usages were updated to use `ThemeName` instead of `string`, too.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
[Internal link](https://github.com/elastic/security-team/issues/10820)
to the feature details
This PR includes next improvements and fixes
### Improvements
1. Add information tooltip for `Status`, `Severity`, `Author`,
`Integrations` and `Actions` column headers. [Figma
link](https://www.figma.com/design/BD9GZZz6y8pfSbubAt5H2W?node-id=2579-182863#1094946220)
https://github.com/user-attachments/assets/8de91149-8b47-4dc1-8a6c-853c9e428522
### Fixes
1. Migration rules page flickering/reloading on filter updates. Make
sure that we show loading indicator for the table only when rules data
is being fetched:
https://github.com/user-attachments/assets/ff24fd50-c286-46a6-a850-9d12d3a01993
2. Make sure that we split translation tab equally between original and
translated query code block components.
https://github.com/user-attachments/assets/c1214f2c-e0a9-4add-82e6-4296458ce7f9
To reproduce this issue you need to add a splunk rule with the long one
line query. For example
> tag=watchlist NOT sourcetype=stash | eval
risk_object=case(isnotnull(user),user,isnotnull(src_user),src_user,isnotnull(dest),dest,isnotnull(src),src,1=1,host)
| eval
risk_object_type=case(isnotnull(user),"user",isnotnull(src_user),"user",isnotnull(dest),"system",isnotnull(src),"system",1=1,"system")
| eval risk_score=if(eventtype="website_watchlist",50,null()) | eval
suppression_value=sourcetype."|".risk_object | `get_event_id` | table
_raw,event_id,host,source,sourcetype,src,dest,dvc,src_user,user
> [!NOTE]
> This feature needs `siemMigrationsEnabled` experimental flag enabled
to work.
## Summary
This PR prevents redundant Discover requests in ES|QL mode for the
following scenarios:
- Creating a new Discover session.
- Saving the current Discover session.
- Loading a saved Discover session.
- Toggling the Unified Histogram chart visibility.
It does so by addressing several underlying state related issues that
were triggering the redundant requests:
- Skipping the initial emission of `currentSuggestionContext` on Unified
Histogram mount, which immediately triggered a second fetch.
- Treating the Unified Histogram `table` prop the same as other props
which affect Lens suggestions (data view, query, columns), and deferring
updates to it until result fetching completes to avoid unnecessary
suggestion updates.
- Removing all auto-fetching behaviour from Unified Histogram and
instead relying solely on the consumer to control when fetching should
occur (including the initial fetch).
Resolves#165192.
### Checklist
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: Matthias Wilhelm <matthias.wilhelm@elastic.co>
Resolves https://github.com/elastic/response-ops-team/issues/251
## Note
This PR includes some saved object schema changes that I will pull out
into their own separate PR in order to perform an intermediate release.
I wanted to make sure all the schema changes made sense in the overall
context of the PR before opening those separate PRs.
Update: PR for intermediate release here:
https://github.com/elastic/kibana/pull/203184 (Merged)
## Summary
Adds ability to run actions for backfill rule runs.
- Updates schedule backfill API to accept `run_actions` parameter to
specify whether to run actions for backfill.
- Schedule API accepts any action where `frequency.notifyWhen ===
'onActiveAlert'`. If a rule has multiple actions where some are
`onActiveAlert` and some are `onThrottleInterval`, the invalid actions
will be stripped and a warning returned in the schedule response but
valid actions will be scheduled.
- Connector IDs are extracted and stored as references in the ad hoc run
params saved object
- Any actions that result from a backfill task run are scheduled as low
priority tasks
## To Verify
1. Create a detection rule. Make sure you have some past data that the
rule can run over in order to generate actions. Make sure you add
actions to the rule. For testing, I added some conditional actions so I
could see actions running only on backfill runs using
`kibana.alert.rule.execution.type: "manual"`. Create actions with and
without summaries.
2. Schedule a backfill either directly via the API or using the
detection UI. Verify that actions are run for the backfill runs that
generate alerts.
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
* Added a few transforms to simplify package paths.
* Fixed typo causing `.mdx` files to not be processed when replacing
references.
* Added preliminary support for `--healthcheck` (to check for broken
references to files and links).
Closes: #205054
**Description**
EQL Query text area is missing accessible label in create new rule panel
of type event correlation
**Preconditions**
Security -> Rules->Detection Rules(SIEM) ->create rule is open
**Steps to reproduce**
1. Click on Event correlation type for rule type
2. Run axe-core on the page
3. Notice the critical a11y violation error for EQL query text area
**Changes made**
1. Added `aria-label` attribute for mentioned places
**Screen**
<img width="897" alt="image"
src="https://github.com/user-attachments/assets/7bcf592b-03d7-407a-aa94-3506d535c161"
/>
**Fixes: https://github.com/elastic/kibana/issues/202715**
**Fixes: https://github.com/elastic/kibana/issues/204714**
## Summary
This PR makes inconsistent/wrong rule's look-back duration prominent for
a user. It falls back to a default 1 minute value in rule upgrade
workflow.
## Details
### Negative/wrong `lookback` problem
There is a difference between rule schedule value in a saved object and
value represented to users
- Saved object (and rule management API) has `interval`, `from` and `to`
fields representing rule schedule. `interval` shows how often a rule
runs in task runner. `from` and `to` stored in date math format like
`now-10m` represent a date time range used to fetch source events. Task
manager strives to run rules exactly every `interval` but it's not
always possible due to multiple reasons like system load and various
delays. To avoid any gaps to appear `from` point in time usually stands
earlier than current time minus `interval`, for example `interval` is
`10 minutes` and `from` is `now-12m` meaning rule will analyze events
starting from 12 minutes old. `to` represents the latest point in time
source events will be analyzed.
- Diffable rule and UI represent rule schedule as `interval` and
`lookback`. Where `interval` is the same as above and `lookback` and a
time duration before current time minus `interval`. For example
`interval` is `10 minutes` and lookback is `2 minutes` it means a rule
will analyzing events starting with 12 minutes old until the current
moment in time.
Literally `interval`, `from` and `to` mean a rule runs every `interval`
and analyzes events starting from `from` until `to`. Technically `from`
and `to` may not have any correlation with `interval`, for example a
rule may analyze one year old events. While it's reasonable for manual
rule runs and gap remediation the same approach doesn't work well for
usual rule schedule. Transformation between `interval`/`from`/`to` and
`interval`/`lookback` works only when `to` is equal the current moment
in time i.e. `now`.
Rule management APIs allow to set any `from` and `to` values resulting
in inconsistent rule schedule. Transformed `interval`/`lookback` value
won't represent real time interval used to fetch source events for
analysis. On top of that negative `lookback` value may puzzle users on
the meaning of the negative sign.
### Prebuilt rules with `interval`/`from`/`to` resulting in negative
`lookback`
Some prebuilt rules have such `interval`, `from` and `to` field values
thatnegative `lookback` is expected, for example `Multiple Okta Sessions
Detected for a Single User`. It runs every `60 minutes` but has `from`
field set to `now-30m` and `to` equals `now`. In the end we have
`lookback` equals `to` - `from` - `interval` = `30 minutes` - `60
minutes` = `-30 minutes`.
Our UI doesn't handle negative `lookback` values. It simply discards a
negative sign and substitutes the rest for editing. In the case above
`30 minutes` will be suggested for editing. Saving the form will result
in changing `from` to `now-90m`
<img width="1712" alt="image"
src="https://github.com/user-attachments/assets/05519743-9562-4874-8a73-5596eeccacf2"
/>
### Changes in this PR
This PR mitigates rule schedule inconsistencies caused by `to` fields
not using the current point in time i.e. `now`. The following was done
- `DiffableRule`'s `rule_schedule` was changed to have `interval`,
`from` and `to` fields instead of `interval` and `lookback`
- `_perform` rule upgrade API endpoint was adapted to the new
`DIffableRule`'s `rule_schedule`
- Rule upgrade flyout calculates and shows `interval` and `lookback` in
Diff View, readonly view and field form when `lookback` is non-negative
and `to` equals `now`
- Rule upgrade flyout shows `interval`, `from` and `to` in Diff View,
readonly view and field form when `to` isn't equal `now` or calculated
`lookback` is negative
- Rule upgrade flyout shows a warning when `to` isn't equal `now` or
calculated `lookback` is negative
- Rule upgrade flyout's JSON Diff shows `interval` and `lookback` when
`lookback` is non-negative and `to` equals `now` and shows `interval`,
`from` and `to` in any other case
- Rule details page shows `interval`, `from` and `to` in Diff View,
readonly view and field form when `to` isn't equal `now` or calculated
`lookback` is negative
- `maxValue` was added to `ScheduleItemField` to have an ability to
restrict input at reasonable values
## Screenshots
- Rule upgrade workflow (negative look-back)
<img width="2558" alt="Screenshot 2025-01-02 at 13 16 59"
src="https://github.com/user-attachments/assets/b8bf727f-11ca-424f-892b-b024ba7f847a"
/>
<img width="2553" alt="Screenshot 2025-01-02 at 13 17 20"
src="https://github.com/user-attachments/assets/9f751ea4-0ce0-4a23-a3b7-0a16494d957e"
/>
<img width="2558" alt="Screenshot 2025-01-02 at 13 18 24"
src="https://github.com/user-attachments/assets/6908ab02-4011-4a6e-85ce-e60d5eac7993"
/>
- Rule upgrade workflow (positive look-back)
<img width="2555" alt="Screenshot 2025-01-02 at 13 19 12"
src="https://github.com/user-attachments/assets/06208210-c6cd-4842-8aef-6ade5d13bd36"
/>
<img width="2558" alt="Screenshot 2025-01-02 at 13 25 31"
src="https://github.com/user-attachments/assets/aed38bb0-ccfb-479a-bb3b-e5442c518e63"
/>
- JSON view
<img width="2559" alt="Screenshot 2025-01-02 at 13 31 37"
src="https://github.com/user-attachments/assets/07575a81-676f-418e-8b98-48eefe11ab00"
/>
- Rule details page
<img width="2555" alt="Screenshot 2025-01-02 at 13 13 16"
src="https://github.com/user-attachments/assets/e977b752-9d50-4049-917a-af2e8e3f0dfe"
/>
<img width="2558" alt="Screenshot 2025-01-02 at 13 14 10"
src="https://github.com/user-attachments/assets/06d6f477-5730-48ca-a240-b5e7592bf173"
/>
## How to test?
- Ensure the `prebuiltRulesCustomizationEnabled` feature flag is enabled
- Allow internal APIs via adding `server.restrictInternalApis: false` to
`kibana.dev.yaml`
- Clear Elasticsearch data
- Run Elasticsearch and Kibana locally (do not open Kibana in a web
browser)
- Install an outdated version of the `security_detection_engine` Fleet
package
```bash
curl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 2023-10-31" -d '{"force":true}' http://localhost:5601/kbn/api/fleet/epm/packages/security_detection_engine/8.14.1
```
- Install prebuilt rules
```bash
curl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 1" -d '{"mode":"ALL_RULES"}' http://localhost:5601/kbn/internal/detection_engine/prebuilt_rules/installation/_perform
```
- Set "inconsistent" rule schedule for `Suspicious File Creation via
Kworker` rule by running a query below
```bash
curl -X PATCH --user elastic:changeme -H "Content-Type: application/json" -H "elastic-api-version: 2023-10-31" -H "kbn-xsrf: 123" -d '{"rule_id":"ae343298-97bc-47bc-9ea2-5f2ad831c16e","interval":"10m","from":"now-5m","to":"now-2m"}' http://localhost:5601/kbn/api/detection_engine/rules
```
- Open rule upgrade flyout for `Suspicious File Creation via Kworker`
rule
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
`cytoscape` released yet another semver breaking change, affecting
typing once more in a couple of places. They now have their own
`index.d.ts` file for Typescript support, but this causes some issues as
they don't always have the same type name as the `@types` version. Had
to fix in a couple of places, both in APM and in ML codebases.
Supersedes #206498
## How to test
* This PR passes the CI with no typing issues, or failed tests
* Going to APM or ML pages with cytoscape being loaded shows no
regressions.
## Summary
Addresses https://github.com/elastic/kibana/issues/202079
Updates the existing import and export rule test plans to include front
end tests as well as more exhaustive coverage of the prebuilt rule
customization milestone 3 epic
## Summary
Epic: https://github.com/elastic/security-team/issues/7998
In this PR we're breaking out the `timeline` and `notes` features into
their own feature privilege definition. Previously, access to both
features was granted implicitly through the `siem` feature. However, we
found that this level of access control is not sufficient for all
clients who wanted a more fine-grained way to grant access to parts of
security solution.
In order to break out `timeline` and `notes` from `siem`, we had to
deprecate it feature privilege definition for. That is why you'll find
plenty of changes of `siem` to `siemV2` in this PR. We're making use of
the feature privilege's `replacedBy` functionality, allowing for a
seamless migration of deprecated roles.
This means that roles that previously granted `siem.all` are now granted
`siemV2.all`, `timeline.all` and `notes.all` (same for `*.read`).
Existing users are not impacted and should all still have the correct
access. We added tests to make sure this is working as expected.
Alongside the `ui` privileges, this PR also adds dedicated API tags.
Those tags haven been added to the new and previous version of the
privilege definitions to allow for a clean migration:
```mermaid
flowchart LR
subgraph v1
A(siem) --> Y(all)
A --> X(read)
Y -->|api| W(timeline_write / timeline_read / notes_read / notes_write)
X -->|api| V(timeline_read /notes_read)
end
subgraph v2
A-->|replacedBy| C[siemV2]
A-->|replacedBy| E[timeline]
A-->|replacedBy| G[notes]
E --> L(all)
E --> M(read)
L -->|api| N(timeline_write / timeline_read)
M -->|api| P(timeline_read)
G --> Q(all)
G --> I(read)
Q -->|api| R(notes_write / notes_read)
I -->|api| S(notes_read)
end
```
### Visual changes
#### Hidden/disabled elements
Most of the changes are happening "under" the hood and are only
expressed in case a user has a role with `timeline.none` or
`notes.none`. This would hide and/or disable elements that would usually
allow them to interact with either timeline or the notes feature (within
timeline or the event flyout currently).
As an example, this is how the hover actions look for a user with and
without timeline access:
| With timeline access | Without timeline access |
| --- | --- |
| <img width="616" alt="Screenshot 2024-12-18 at 17 22 49"
src="https://github.com/user-attachments/assets/a767fbb5-49c8-422a-817e-23e7fe1f0042"
/> | <img width="724" alt="Screenshot 2024-12-18 at 17 23 29"
src="https://github.com/user-attachments/assets/3490306a-d1c3-41aa-af5b-05a1dd804b47"
/> |
#### Roles
Another visible change of this PR is the addition of `Timeline` and
`Notes` in the edit-role screen:
| Before | After |
| ------- | ------ |
| <img width="746" alt="Screenshot 2024-12-12 at 16 31 43"
src="https://github.com/user-attachments/assets/20a80dd4-c214-48a5-8c6e-3dc19c0cbc43"
/> | <img width="738" alt="Screenshot 2024-12-12 at 16 32 53"
src="https://github.com/user-attachments/assets/afb1eab4-1729-4c4e-9f51-fddabc32b1dd"
/> |
We made sure that for migrated roles that hard `security.all` selected,
this screen correctly shows `security.all`, `timeline.all` and
`notes.all` after the privilege migration.
#### Timeline toast
There are tons of places in security solution where `Investigate / Add
to timeline` are shown. We did our best to disable all of these actions
but there is no guarantee that this PR catches all the places where we
link to timeline (actions). One layer of extra protection is that the
API endpoints don't give access to timelines to users without the
correct privileges. Another one is a Redux middleware that makes sure
timelines cannot be shown in missed cases. The following toast will be
shown instead of the timeline:
<img width="354" alt="Screenshot 2024-12-19 at 10 34 23"
src="https://github.com/user-attachments/assets/1304005e-2753-4268-b6e7-bd7e22d8a1e3"
/>
### Changes to predefined security roles
All predefined security roles have been updated to grant the new
privileges (in ESS and serverless). In accordance with the migration,
all roles with `siem.all` have been assigned `siemV2.all`,
`timeline.all` and `notes.all` (and `*.read` respectively).
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: PhilippeOberti <philippe.oberti@elastic.co>
Co-authored-by: Steph Milovic <stephanie.milovic@elastic.co>
This adds support for the new selector syntax to the log source profile
heuristics. It will only match when index name expression exclusively
contains implicit or explicit `data` selectors.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Currently, the `semantic_text` field supports a default `inference_id`,
meaning users are not required to explicitly select an inference
endpoint during mapping. However, a bug has been identified: if the
`Select inference Id` popover is not opened, the `inference_id` field
property remains as an empty string. This causes Elasticsearch (ES) to
throw an error, as it requires a value to be present if the property is
defined.
To address this issue, the proposed solution is to remove the
`inference_id` property from the `semantic_text` field during field
mapping if its value is empty.
### Screen Recording
https://github.com/user-attachments/assets/e8d8d471-7ff2-493e-8872-e42838579d44
---------
Co-authored-by: Matthew Kime <matt@mattki.me>
## Summary
resolves https://github.com/elastic/kibana/issues/105692
This PR adds a pre response handler that sets a warning header if the
requested endpoint is deprecated.
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.
Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Added createdBy and updatedBy fields in summary documents !!
This will make easier to identify which user have added the SLO and
which user last updated the SLO. It's especially helpful where there are
100s of SLOs defined.
<img width="1728" alt="image"
src="https://github.com/user-attachments/assets/ee7bb4d4-a8ea-40c4-8d91-06c32c9b0ba6"
/>
---------
Co-authored-by: Kevin Delemme <kdelemme@gmail.com>
Co-authored-by: Kevin Delemme <kevin.delemme@elastic.co>
Fixes https://github.com/elastic/security-team/issues/11357
## Summary
In this PR we use cases local storage to preserve the selection of
ordering in the user activity on the cases detail page.
Initially, I was going to save the whole `UserActivityParams` on local
storage but ultimately decided against it just to preserve the defaults
like "selected tab" or "page".
## Summary
This PR improves upon the Universal entity definition and entity store
work being done to support Asset Inventory by introducing a flag
`dynamic` to the definition.
The entity store uses an enrich policy in order to retain observed data
that falls outside of a `lookbackPeriod` used by the transform that runs
the aggregations on the source fields.
Normally, we have to specify a retention strategy per each field defined
in an entity definition. However, for universal entities, (some of) the
fields are dynamically generated based on the JSON extractor pipeline
processor, which means we cannot define which strategy to use in the
definition itself.
To account for this, when `dynamic` is set to `true`, we run an extra
ingest pipeline step to process _any field which does not show up in the
entity definition_ (ie, has been dynamically generated). At the moment,
this pipeline step uses a strategy that always picks the latest value,
although int he future, this might need to be configurable, mimicking
the ability to choose strategies for "static" fields.
See this
[doc](https://docs.google.com/document/d/1D8xDtn3HHP65i1Y3eIButacD6ZizyjZZRJB7mxlXzQY/edit?tab=t.0#heading=h.9fz3qtlfzjg7)
for more details and [this
Figma](https://www.figma.com/board/17dpxrztlM4O120p9qMcNw/Entity-descriptions?node-id=0-1&t=JLcB84l9NxCnudAs-1)
for information regarding Entity Store architecture.
## How to test:
### Setup
1. Ensure the default Security Data View exists by navigating to some
Security solution UI.
2. Set up the `entity.keyword` builder pipeline
* Add it to an index that matches any of the default index patterns in
the security data view (eg: `logs-store`)
* Make sure and ingested doc contains both `event.ingested` and
`@timestamp`.
* Easiest way is to add `set` processors to the builder pipeline.
3. Because of the async nature of the field retention process, it is
recommended to change some of the default values (explained below)
4. Enable `debugging` by adding
`xpack.securitySolution.entityAnalytics.entityStore.developer.pipelineDebugMode:
true` to your `kibana.dev.yml`
5. Enable the `assetInventoryStoreEnabled` FF:
```
xpack.securitySolution.enableExperimental:
- assetInventoryStoreEnabled
```
### Interacting with the store
In Kibana dev tools:
#### Phase 1
1. `POST` some of the example docs to the `logs-store` index
2. Confirm the `entity.keyword` field is being added by the builder
pipeline via `GET logs-store/_search`.
3. Initialise the universal entity engine via: `POST
kbn:/api/entity_store/engines/universal/init {}`
* In order to properly test field retention, it's advisable to reduce
the `lookbackPeriod` setting, which means some of the docs in the index
might fall out of the window if it takes too long to initialize the
engine. Any docs posted when the engine is running should be picked up.
* Note that using the UI does not work, as we've specifically removed
the Universal engine from the normal Entity Store workflow
4. Check the status of the store is `running` via `GET
kbn:/api/entity_store/status`
5. Check that the transform has ran by querying the store index: `GET
.entities.v1.latest.security_universal*/_search`
* There should be one entity per `related.entity` found in the source
index
* The fields in the JSON string in `entities.keyword` should appear as
fields in the target documents
* There should also be a `debug` field and potentially a `historical`
field, if enough time has passed for the enrich policy to run. These are
normally hidden, but show up when in `debug mode`.
#### Phase 2
1. Wait some time (the `INTERVAL` constant) for the enrich policy to
populate the `.enrich` indices with the latest data from the store index
* Ideally, this will mean that any docs in the source index now fall
outside of `lookbackPeriod` of the transform.
* Alternatively, you can manually run the enrich poly via: `PUT
/_enrich/policy/entity_store_field_retention_universal_default_v1.0.0/_execute`.
* It's also possible to update the source docs' timestamps and
`event.ingested` to ensure they're outside the `lookbackPeriod`
3. `POST` a new doc to the source index (eg: `logs-store`)
* The new doc should either have a new, not yet observed property in
`entities.metadata`, or the same fields but with different, new values.
4. Query the store index again.
* The entity in question should now reflect the new changes _but
preserve the old data too!_
* Existing fields should have been updated to new values
* New fields should have been `recursively` merged. Ie, nested fields
should not be an issue.
* The `historical` field should show the "previous state" of the entity
doc. This is useful to confirm that a field's value is, in fact, the
"latest" value, whether that comes from a new doc that falls in the
lookback window of the transform, or from this `historical` "cache".
### Code
#### Default values:
* in
[`server/lib/entity_analytics/entity_store/entity_definition/universal.ts#L75-L76`](6686d57ce5/x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/entity_store/entity_definitions/entity_descriptions/universal.ts (L75-L76)):
* Add the following fields to `settings`:
```ts
{ frequency: '2s', lookbackPeriod: '1m', syncDelay: '2s'}
```
* in
[`server/lib/entity_analytics/entity_store/task/constants.ts#L11-L13`](6686d57ce5/x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/entity_store/task/constants.ts (L11-L13))
* Change the following defaults:
```ts
export const INTERVAL = '1m';
export const TIMEOUT = '30s';
```
#### Ingest pipeline
<details>
<summary>Pipeline</summary>
```js
PUT _ingest/pipeline/entities-keyword-builder
{
"description":"Serialize entities.metadata into a keyword field",
"processors":[
{
"set": {
"field": "event.ingested",
"value": "{{_ingest.timestamp}}"
}
},
{
"set": {
"field": "@timestamp",
"value": "{{_ingest.timestamp}}"
}
},
{
"script":{
"lang":"painless",
"source":"""
String jsonFromMap(Map map) {
StringBuilder json = new StringBuilder("{");
boolean first = true;
for (entry in map.entrySet()) {
if (!first) {
json.append(",");
}
first = false;
String key = entry.getKey().replace("\"", "\\\"");
Object value = entry.getValue();
json.append("\"").append(key).append("\":");
if (value instanceof String) {
String escapedValue = ((String) value).replace("\"", "\\\"").replace("=", ":");
json.append("\"").append(escapedValue).append("\"");
} else if (value instanceof Map) {
json.append(jsonFromMap((Map) value));
} else if (value instanceof List) {
json.append(jsonFromList((List) value));
} else if (value instanceof Boolean || value instanceof Number) {
json.append(value.toString());
} else {
// For other types, treat as string
String escapedValue = value.toString().replace("\"", "\\\"").replace("=", ":");
json.append("\"").append(escapedValue).append("\"");
}
}
json.append("}");
return json.toString();
}
String jsonFromList(List list) {
StringBuilder json = new StringBuilder("[");
boolean first = true;
for (item in list) {
if (!first) {
json.append(",");
}
first = false;
if (item instanceof String) {
String escapedItem = ((String) item).replace("\"", "\\\"").replace("=", ":");
json.append("\"").append(escapedItem).append("\"");
} else if (item instanceof Map) {
json.append(jsonFromMap((Map) item));
} else if (item instanceof List) {
json.append(jsonFromList((List) item));
} else if (item instanceof Boolean || item instanceof Number) {
json.append(item.toString());
} else {
// For other types, treat as string
String escapedItem = item.toString().replace("\"", "\\\"").replace("=", ":");
json.append("\"").append(escapedItem).append("\"");
}
}
json.append("]");
return json.toString();
}
def metadata = jsonFromMap(ctx['entities']['metadata']);
ctx['entities']['keyword'] = metadata;
"""
}
}
]
}
```
</details>
<details>
<summary>Index template</summary>
```js
PUT /_index_template/entity_store_index_template
{
"index_patterns":[
"logs-store"
],
"template":{
"settings":{
"index":{
"default_pipeline":"entities-keyword-builder"
}
},
"mappings":{
"properties":{
"@timestamp":{
"type":"date"
},
"message":{
"type":"text"
},
"event":{
"properties":{
"action":{
"type":"keyword"
},
"category":{
"type":"keyword"
},
"type":{
"type":"keyword"
},
"outcome":{
"type":"keyword"
},
"provider":{
"type":"keyword"
},
"ingested":{
"type": "date"
}
}
},
"related":{
"properties":{
"entity":{
"type":"keyword"
}
}
},
"entities":{
"properties":{
"metadata":{
"type":"flattened"
},
"keyword":{
"type":"keyword"
}
}
}
}
}
}
}
```
</details>
<details>
<summary>Example source docs</summary>
#### Phase 1:
```js
POST /logs-store/_doc/
{
"related":{
"entity":[
"test-id"
]
},
"entities":{
"metadata":{
"test-id":{
"okta":{
"foo": {
"baz": {
"qux": 1
}
}
},
"cloud": {
"super": 123
}
}
}
}
}
```
```js
POST /logs-store/_doc/
{
"related":{
"entity":[
"test-id"
]
},
"entities":{
"metadata":{
"test-id":{
"cloud":{
"host": "me"
}
}
}
}
}
```
#### Phase 2:
```js
POST /logs-store/_doc/
{
"related":{
"entity":[
"test-id"
]
},
"entities":{
"metadata":{
"test-id":{
"cloud":{
"host": "me",
"super": 1111111,
},
"okta":{
"foo": {
"baz": {
"qux": 99,
"hello": "world"
},
"hello": "world"
},
"hello": "world"
}
}
}
}
}
```
</details>
Closes#204116
## Summary
fix:
o11y assistant Error, when using the model (llama 3.2) the stream get
closed in the middle and fails with an error related to the title
generation
This PR contains the following updates:
| Package | Type | Update | Change |
|---|---|---|---|
| [oas](https://togithub.com/readmeio/oas)
([source](https://togithub.com/readmeio/oas/tree/HEAD/packages/oas)) |
dependencies | patch | [`^25.2.0` ->
`^25.2.1`](https://renovatebot.com/diffs/npm/oas/25.2.0/25.2.1) |
---
### Release Notes
<details>
<summary>readmeio/oas (oas)</summary>
###
[`v25.2.1`](https://togithub.com/readmeio/oas/compare/oas@25.2.0...oas@25.2.1)
[Compare
Source](https://togithub.com/readmeio/oas/compare/oas@25.2.0...oas@25.2.1)
</details>
---
### Configuration
📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR has been generated by [Renovate
Bot](https://togithub.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40MjUuMSIsInVwZGF0ZWRJblZlciI6IjM3LjQyNS4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJUZWFtOlNlY3VyaXR5LVNjYWxhYmlsaXR5IiwiYmFja3BvcnQ6YWxsLW9wZW4iLCJyZWxlYXNlX25vdGU6c2tpcCJdfQ==-->
Co-authored-by: elastic-renovate-prod[bot] <174716857+elastic-renovate-prod[bot]@users.noreply.github.com>
This PR contains the following updates:
| Package | Update | Change |
|---|---|---|
| docker.elastic.co/wolfi/chainguard-base | digest | `dd66bee` ->
`ea157dd` |
---
### Configuration
📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR has been generated by [Renovate
Bot](https://togithub.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40MjUuMSIsInVwZGF0ZWRJblZlciI6IjM3LjQyNS4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJUZWFtOk9wZXJhdGlvbnMiLCJiYWNrcG9ydDpza2lwIiwicmVsZWFzZV9ub3RlOnNraXAiXX0=-->
Co-authored-by: elastic-renovate-prod[bot] <174716857+elastic-renovate-prod[bot]@users.noreply.github.com>
Part of https://github.com/elastic/kibana/issues/203664
## Summary
EUI added `behindText` vis colors to the euiTheme. Replacing here
`euiThemeVars` with the new vis colors.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Extends the Observability AI Assistant's evaluation framework to create
the first set of tests aimed at evaluating the performance of the
Investigation App's AI root cause analysis integration.
To execute tests, please consult the
[README](https://github.com/elastic/kibana/pull/204634/files#diff-4823a154e593051126d3d5822c88d72e89d07f41b8c07a5a69d18281c50b09adR1).
Note the prerequisites and the Kibana & Elasticsearch configuration.
Further evolution
--
This PR is the first MVP of the evaluation framework. A (somewhat light)
[meta issue](https://github.com/elastic/kibana/issues/205670) exists for
our continued work on this project, and will be added to over time.
Test data and fixture architecture
--
Logs, metrics, and traces are indexed to
[edge-rca](https://studious-disco-k66oojq.pages.github.io/edge-rca/).
Observability engineers can [create an oblt-cli
cluster](https://studious-disco-k66oojq.pages.github.io/user-guide/cluster-create-ccs/)
configured for cross cluster search against edge-rca as the remote
cluster.
When creating new testing fixtures, engineers will utilize their
oblt-cli cluster to create rules against the remote cluster data. Once
alerts are triggered in a failure scenario, the engineer can choose to
archive the alert data to utilize as a test fixture.
Test fixtures are added to the `investigate_app/scripts/load/fixtures`
directory for use in tests.
When execute tests, the fixtures are loaded into the engineer's oblt-cli
cluster, configured for cross cluster search against edge-rca. The local
alert fixture and the remote demo data are utilized together to replay
root cause analysis and execute the test evaluations.
Implementation
--
Creates a new directory `scripts`, to house scripts related to setting
up and running these tests. Here's what each directory does:
## scripts/evaluate
1. Extends the evaluation script from
`observability_ai_assistant_app/scripts/evaluation` by creating a
[custom Kibana
client](https://github.com/elastic/kibana/pull/204634/files#diff-ae05b2a20168ea08f452297fc1bd59310c69ac3ea4651da1f65cd9fa93bb8fe9R1)
with RCA specific methods. The custom client is [passed to the
Observability AI Assistant's
`runEvaluations`](https://github.com/elastic/kibana/pull/204634/files#diff-0f2d3662c01df8fbe7d1f19704fa071cbd6232fb5f732b313e8ba99012925d0bR14)
script an[d invoked instead of the default Kibana
Client](https://github.com/elastic/kibana/pull/204634/files#diff-98509a357e86ea5c5931b1b46abc72f76e5304439430358eee845f9ad57f63f1R54).
2. Defines a single, MVP test in `index.spec.ts`. This test find a
specific alert fixture designated for that test, creates an
investigation for that alert with a specified time range, and calls the
root cause analysis api. Once the report is received back from the api,
a prompt is created for the evaluation framework with details of the
report. The evaluation framework then judges how well the root cause
analysis api performed against specified criteria.
## scripts/archive
1. Utilized when creating new test fixtures, this script will easily
archive observability alerts data for use as a fixture in a feature test
## scripts/load
1. Loads created testing fixtures before running the test.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Dario Gieselaar <d.gieselaar@gmail.com>
Closes https://github.com/elastic/kibana/issues/204992
## Summary
Callout for single edit data retention (opened from data stream details
panel):
<img width="1446" alt="Screenshot 2025-01-15 at 13 29 29"
src="https://github.com/user-attachments/assets/c415e634-4b39-43d3-b1ae-8a1de55cb144"
/>
For reference, this is the callout for bulk edit data retention (exists
from before this PR):
<img width="1446" alt="Screenshot 2025-01-15 at 13 26 08"
src="https://github.com/user-attachments/assets/6d167f94-9882-4b48-b1f9-20d26e9bdea7"
/>
**How to test:**
1. Start Es and Kibana
2. Go to Index Management -> Data streams and click on one of the data
streams.
3. Click on the "Manage" button and edit data retention.
4. Decrease the data retention period and verify that the callout
message is correct.
5. Also, verify that the callout message in the bulk edit data retention
modal is still the same.
This adds an e2e test for [the Ensemble
workflow](https://github.com/elastic/ensemble/actions/workflows/nightly.yml)
to cover stack installation part of the OTel K8S quickstart flow.
Besides that I've replaced the retry logic for K8S EA and Auto Detect
flow with a simple timeouts to workaround the missing data issue on the
CTA pages (host details and k8s dashboard) after finishing the
onboarding flow. I've also simplified assertions on the CTA pages.
