### Background
In some scenarios we need to perform some setup at bootstrap time, this
setup could affect following scenarios.
Take for example
[failed_logs](https://github.com/elastic/kibana/blob/main/packages/kbn-apm-synthtrace/src/scenarios/failed_logs.ts)
scenario where we create a pipeline that will do some checks in
`log.level` property, if we try to run an scenario after that one we
will enter into some issues.
### Changes
This PR aims to introduce a `teardown` setup for scenarios where we
could undo the changes done at `bootstrap` time.
## Summary
If you enable streams (which creates `logs`) and then try to create
`logs.child.grandchild` but `logs.child` already exists as either an
index or an unwired (Classic) stream, then we end up in a weird state
where `logs.child.grandchild` gets created as a wired child but then the
request fails as it tries to turn the unwired stream into a wired
stream.
This PR adds a step that asserts that there are no such conflicts in the
hierarchy before proceeding.
It also adds a check to ensure Streams are enabled before allowing the
creation of any streams, as well as blocking the creation of a root
stream that isn't `logs`.
Finally, there is some minor improvements to error handling for when a
data stream isn't found and error messages.
fix https://github.com/elastic/kibana/issues/167676 for 9.0 promotion
steps taken:
1. Start Node01:
`yarn es snapshot --version=8.19.0 --data-archive
src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip
--base-path .es/node01`
2. Start Node02:
`yarn es snapshot --version=8.19.0 --data-archive
src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip
--base-path .es/node02`
3. Wait for both nodes to start and be ready
4. Stop both ES nodes
5. change into the directory for node01:
`cd .es/node01/8.19.0`
6. Remove (or move) the outdated archive:
`rm
../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip`
7. zip the updated data and store as the original test filename in the
test directory:
`zip -r
../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip
data -x "*/\.*"`
Repeat these steps for node02:
8. `cd ../../../`
9. `cd .es/node02/8.19.0`
10. `rm
../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip`
11. `zip -r
../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip
data -x "*/\.*"`
12. `cd ../../../`
13. `git commit -m "Updated archives" && git push`
### Identify risks
Data archives have to be updated for every major.last before a new major
is released.
Should updating the archives not resolve test failures, there may be a
bug elsewhere in the stack. If so, carefully inspect the error logs to
determine the cause.
## Summary
Adds audit logging for successful calls to these API routes:
- Start/Stop migration
- Create migration
- Update Rule
- Install Rule
- Retrieve migration results.
- Uploaded Macro/Lookup
- Retrieved Macro/Lookup
Tested it manually by going through the workflow with audit logging
enabled:
Enable:
`xpack.security.audit.enabled: true`
Results:
```
{"event":{"action":"siem_migration_created","category":["database"],"type":["creation"],"outcome":"success"},"user":{"name":"elastic","roles":["superuser"]},"kibana":{"space_id":"default"},"trace":{"id":"00885dd4-7fd9-45fe-9a0b-2173adcac4ad"},"client":{"ip":"127.0.0.1"},"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-01-22T15:05:12.875+01:00","message":"User created a new SIEM migration with [id=cd9552ce-05c8-4893-b659-b5a5ed9325d9","log":{"level":"INFO","logger":"plugins.security.audit.ecs"},"process":{"pid":91324,"uptime":540.063456227},"transaction":{"id":"ee4606116856393c"}}
{"event":{"action":"siem_migration_started","category":["database"],"type":["start"],"outcome":"success"},"user":{"name":"elastic","roles":["superuser"]},"kibana":{"space_id":"default"},"trace":{"id":"e852b328-9e53-4c4d-b8ca-b8fa2b76383d"},"client":{"ip":"127.0.0.1"},"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-01-22T15:11:36.569+01:00","message":"User started an existing SIEM migration with [id=3805f79e-123c-4962-b22b-8ddf365cdd89]","log":{"level":"INFO","logger":"plugins.security.audit.ecs"},"process":{"pid":5438,"uptime":62.828177986},"transaction":{"id":"0ac652c8f722f1c4"}}
{"event":{"action":"siem_migration_stopped","category":["database"],"type":["end"],"outcome":"success"},"user":{"name":"elastic","roles":["superuser"]},"kibana":{"space_id":"default"},"trace":{"id":"136f3a38-d47d-455d-bca2-aaf38559b20a"},"client":{"ip":"127.0.0.1"},"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-01-22T15:11:05.871+01:00","message":"User stopped an existing SIEM migration with [id=3805f79e-123c-4962-b22b-8ddf365cdd89]","log":{"level":"INFO","logger":"plugins.security.audit.ecs"},"process":{"pid":5438,"uptime":32.12840553},"transaction":{"id":"be379686654f4bdf"}}
{"event":{"action":"siem_migration_updated_rule","category":["database"],"type":["change"],"outcome":"success"},"user":{"id":"u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0","name":"elastic","roles":["superuser"]},"kibana":{"space_id":"default","session_id":"IPgQ3+R8DW9uxx4RQqUx9eZj+D5Es7SGQdcDoM/02l4="},"trace":{"id":"368b31e7-812d-464b-83d8-0e635c7fe5ed"},"client":{"ip":"127.0.0.1"},"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-01-22T15:13:15.827+01:00","message":"User updated a translated detection rule with [id=29pWjpQB_LGnD_bEV66u]","log":{"level":"INFO","logger":"plugins.security.audit.ecs"},"process":{"pid":5438,"uptime":162.09338085},"transaction":{"id":"404b3cb31be3c94f"}}
{"event":{"action":"siem_migration_installed_rule","category":["database"],"type":["creation"],"outcome":"success"},"user":{"id":"u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0","name":"elastic","roles":["superuser"]},"kibana":{"space_id":"default","session_id":"IPgQ3+R8DW9uxx4RQqUx9eZj+D5Es7SGQdcDoM/02l4="},"trace":{"id":"d90396dc-a0d3-4308-b07e-54761b562803"},"client":{"ip":"127.0.0.1"},"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-01-22T15:13:46.709+01:00","message":"User installed a new detection rule through SIEM migration with [id=3805f79e-123c-4962-b22b-8ddf365cdd89]","log":{"level":"INFO","logger":"plugins.security.audit.ecs"},"process":{"pid":5438,"uptime":192.977728866},"transaction":{"id":"4b17bafb9fbf48a4"}}
{"event":{"action":"siem_migration_uploaded_macro","category":["database"],"type":["creation"],"outcome":"success"},"user":{"name":"elastic","roles":["superuser"]},"kibana":{"space_id":"default"},"trace":{"id":"e5fcb9cc-9d27-41cf-a171-13b9faf6078e"},"client":{"ip":"127.0.0.1"},"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-01-23T13:45:18.639+01:00","message":"User uploaded a new macro through SIEM migration with [id=2fc199ef-5bc7-4d87-a349-baeaea662273]","log":{"level":"INFO","logger":"plugins.security.audit.ecs"},"process":{"pid":89625,"uptime":106.911700803},"transaction":{"id":"fc95c131e86b0284"}}
```
## Release Note
Fix Structured log template to use single quotes
### Summary
Currently with a single backslash as escape character the template is
setting up a double quote surrounding it and it causes YAML Exception
```
1 | - kv:
2 | field: message
3 | field_split: "\|"
--------------------------^
YAMLException: unknown escape sequence (3:22)
```
This PR fixes to use single quotes to not break YAML parsing.
## Summary
Fixes#206555
This PR is an attempt to address the `null` bucket issue with `count` in
Lens formula via the `emptyAsNull` flag.
### Checklist
* [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### Risks
This PR introduces potentially some breaking changes, as count `null`
values, in particular coming from shifted computations, as now converted
to `0` if the flag has been enabled.
This change is not news in the code base as other aggs like
`distinct_count` or `value_count` already implements it, but not
`count`.
Apparently no test failed with this change, I've also added new unit
ones to freeze the current behaviour and detect future changes.
---------
Co-authored-by: Marco Vettorello <vettorello.marco@gmail.com>
Co-authored-by: Peter Pisljar <peter.pisljar@gmail.com>
Closes#207005
## Summary
This PR skips flaky tests - 2 of the flaky tests were already skipped so
I skipped the test on the top level (as all the tests will be skipped
anyway it makes more sense to have the whole test skipped)
I tried to fix it and managed to reproduce the flaky behavior only once
out of many runs (the input was not filled but the next check failed -
which was not an expected behavior as it should fail on the previous
step or retry it) - It's super hard to reproduce it. I followed the
steps locally and it worked as expected so it's not an actual issue:
https://github.com/user-attachments/assets/d0d33622-c186-4b31-bcf7-b2c27df330ac
As we plan to refactor the test anyway we should not spend more time on
it so I skipped it for now.
Intermediate releases to change rule or connector schemas in serverless
for #185032
## Summary
An Observability AI Assistant connector is available to be set as action
for Observability rules. When an alert is triggered, a conversation with
the AI assistant will be created sending the initial prompt set by the
user in the rule action. The conversation is then stored and can be
retrieved from the AI Assistant interface. the action is triggered on
any status change of the alert (active, recovered, untracked), creating
a new conversation for each of them using the same initial prompt which
may not be suitable for the 3 cases.
Improvement
The user is able to choose in when the action should be run (active,
recovered, untracked, all),. That would allow the user to specify more
than one AI Assistant action, with a different and more suitable prompt
in each case.
Closes#207734
This PR:
* Changes the behavior of the search form to always include quickstart
flows in search results even when a category is not selected
* Refactors and cleans up the code a bit
https://github.com/user-attachments/assets/e5de7092-2d9f-41be-8d69-25954e5e4bff
## How to test
Make sure that the search works as expected and when clicking on the
cards it leads to the right places.
# Summary
As part of the effort to add missing content for Security APIs, this PR
introduces a few missing request, response, and parameter examples for
Detection Engine Alert and migration APIs.
## Summary
In #189895 we added logic to the `getColorCategories` util function to
handle transpose columns. This mistakenly assumed the first row of the
datatable would include all transposed column ids (i.e.
`${value}---${columnId}`). After closer analysis this case is only
present the in datatable rendering (i.e. `table_basic.tsx`), but in this
context we also have the original non-transposed datatable.
So to simplify this we revert this logic to not care about transposed
datatables. Now the color mappings are correctly assigned across **split
by** columns.
<img width="720" alt="image"
src="https://github.com/user-attachments/assets/c588930e-53b9-409f-a257-2c5be35aaa38"
/>
Fixes#208555
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Release note
Fixes an issue in Lens (#208555) Table where a split-by metric on a
terms rendered incorrect colors in table cells.
---------
Co-authored-by: Marta Bondyra <4283304+mbondyra@users.noreply.github.com>
Co-authored-by: Marco Liberati <dej611@users.noreply.github.com>
## Summary
This PR is a followup to #205990, which removed references to all of the
deprecated/renamed EUI vars in preparation for 9.0. Here, we address
some of the non-critical tech debt related to the EUI refresh, namely
the [removal of static EUI
tokens](https://github.com/elastic/kibana/issues/199715#json-tokens)
from our codebase.
I made every attempt not to change any styles in this PR, except to
simplify CSS to produce an equivalent design. A common example of this
was removing a static `margin` or `padding` declaration referencing
`euiThemeVars.size*`, and swapping it with an equivalent `gutterSize`
prop on the `EuiFlexGroup` container, or with an `align-self` or other
equivalent flexbox directive.
## Screenshots of Areas Affected
The majority of changes here involved the Exception List/Item pages.
I've attached screenshots of their current layout for comparison/review:
<details>
<summary>
<h3>Rule Exceptions Tab</h3>
</summary>
<kbd>
<h3>Before</h3>
<img width="1395" alt="Rule exceptions tab - before"
src="https://github.com/user-attachments/assets/db7a5487-7df3-4a5f-b88e-90ab34784970"
/>
</kbd>
<kbd>
<h3>After</h3>
<img width="1421" alt="Rule exceptions tab - after"
src="https://github.com/user-attachments/assets/77cbdefc-cbec-4b9e-8436-197f2f2f6677"
/>
</kbd>
</details>
<details>
<summary>
<h3>Shared Exception Lists</h3>
</summary>
<kbd>
<img width="1517" alt="Shared Exception Lists"
src="https://github.com/user-attachments/assets/5448dd65-60f7-470c-bf7a-7af75bb914fa"
/>
</kbd>
</details>
<details>
<summary>
<h3>Shared Exception List Details</h3>
</summary>
<kbd>
<img width="1517" alt="Shared Exception List Details"
src="https://github.com/user-attachments/assets/6e03fef6-af58-40bb-98c5-188651a584cc"
/>
</kbd>
</details>
<details>
<summary>
<h3>Threshold Input</h3>
</summary>
<kbd>
<img width="1046" alt="Threshold Input"
src="https://github.com/user-attachments/assets/28738857-6bdf-404f-a790-a9f4e66ff27a"
/>
</kbd>
</details>
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
While working on replacing all `styled-components`with `@emotion/react`
I realized that there is some dead code in the Timelines plugin. Both
the `Overflow` and `ColumnToggle` components used in the `HoverActions`
code exported to other plugins are not used anywhere.
Moreover that `HoverConfig` logic has been deprecated for years now...
I thought about removing all the other hover action components
(`AddToTimeline`, `Copy`, `FilterIn` and `FilterOut`) but these are
currently in used in other plugins (`threat_intelligence` and
`kubernetes_security`) so the effort to replace these are a bit more
involved and will be done in a separate PR
This PR also removes an unused LoadingPanel component.
**No changes to the application are introduced in this PR!**
Fixes: https://github.com/elastic/kibana/issues/195191
Move log threshold rule type params to the new package.
P.S.: I've moved function `validateKQLStringFilter` and test for it in
my previous PR: https://github.com/elastic/kibana/pull/205507
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
**Fixes https://github.com/elastic/kibana/issues/208251**
## Summary
This bug was caused by the local generated MITRE data we have stored in
`x-pack/solutions/security/plugins/security_solution/public/detections/mitre/mitre_tactics_techniques.ts`
having an inconsistency in the way its reference urls were written
compared to the TRADE team's prebuilt rule packages. The trailing
backslash was present in the prebuilt rule packages (and added by
browsers) but not in the url field from the `.json` file we scrape the
MITRE data from in our script.
For example, this is the url from the script:
```
https://attack.mitre.org/techniques/T1078/004
```
and this is the url directly from the rule package:
```
https://attack.mitre.org/techniques/T1078/004/
```
This PR adds a normalization function that adds a trailing backslash to
the comparison string for the diff algorithm if it doesn't already
exist.
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
This removes an override that was applied to the charts theme prior to
updating the `Theme` in `@elastic/charts`. This is no longer needed and
is now using the wrong color.
## Release note
Fixes issue with `Amsterdam` theme where charts render with the
incorrect background color.
fix https://github.com/elastic/kibana/issues/209081
Updates UI copy for the batch reindexing API only supporting indices.
Note to reviewers: There wasn't an entry for the translation id for
French.
---------
Co-authored-by: Jean-Louis Leysens <jloleysens@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Resolves https://github.com/elastic/kibana/issues/208750
## Summary
Removes the hard-coded value in the summarized alerts query, and uses
the max alerts config instead.
### Checklist
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### To verify
1. Set `xpack.alerting.rules.run.alerts.max` in kibana.yml and start
Kibana
2. Create a summary alerting rule
3. Verify that the size in the summary alert query is the same value as
the max alerts config
Closes#209227
## Summary
This PR fixes a failed test: It was not possible to reproduce it locally
but I saw in the failed build that the page (chart) was still loading
when it failed so I added `waitForChartsToLoad` in the places where we
check the charts in the Overview tab.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Close https://github.com/elastic/kibana/issues/209471
# Backport
This will backport the following commits from `8.18` to `main`:
- [[UA] Preserve hidden status of original index during reindexing
(#209512)](https://github.com/elastic/kibana/pull/209512)
<!--- Backport version: 9.6.4 -->
### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)
<!--BACKPORT [{"author":{"name":"Jean-Louis
Leysens","email":"jeanlouis.leysens@elastic.co"},"sourceCommit":{"committedDate":"2025-02-04T13:20:51Z","message":"[UA]
Preserve hidden status of original index during reindexing
(#209512)","sha":"ed5b521cb491ccb4e1491190f221209de1c2a90f","branchLabelMapping":{"^v8.16.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","release_note:skip","Feature:Upgrade
Assistant","backport:skip"],"title":"[UA] Preserve hidden status of
original index during
reindexing","number":209512,"url":"https://github.com/elastic/kibana/pull/209512","mergeCommit":{"message":"[UA]
Preserve hidden status of original index during reindexing
(#209512)","sha":"ed5b521cb491ccb4e1491190f221209de1c2a90f"}},"sourceBranch":"8.18","suggestedTargetBranches":[],"targetPullRequestStates":[]}]
BACKPORT-->
## Summary
Resolves#180590
Since SLO instanceId is provided in all APIs, it will be made required
for sloWithDataResponseSchema and the SLOWithSummaryResponse type.
Checks for the existence of instanceId have either been removed or
changed to check for ALL_VALUE (`*`)
---------
Co-authored-by: Kevin Delemme <kevin.delemme@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
