Show data retention on streams
In case of a policy, the name of the policy is shown (badge is clickable
and leads to the edit page of the policy):
<img width="524" alt="Screenshot 2024-12-12 at 20 57 36"
src="https://github.com/user-attachments/assets/2664b45b-2473-49c4-b1d6-dccb8fe48d43"
/>
In case of DLM, the effect retention is shown:
<img width="532" alt="Screenshot 2024-12-12 at 20 58 42"
src="https://github.com/user-attachments/assets/07ca8086-75e2-45f8-9d71-17bd0a76ebe5"
/>
This is just the display piece, editing retention will be added later
on.
This PR adjusts the base streams data stream settings to use a localized
data stream retention configuration to make it compatible with
serverless.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Fix https://github.com/elastic/kibana/issues/205606
- Re-generate the internal ES|QL documentation using the generation
script (+ human review)
- Add more scenario to the NL-to-ESQL evaluation suite
- Some prompt engineering
- improving the system instructions / functions summary
- add more examples to the summary
- adapt a few opinionated examples for some specific functions
## Evaluation
- average based on 4 runs for each model/branch tuple
- the new tests were locally added to main to run against the same suite
and properly evaluate the difference
| Model | before (main) | after (PR) | delta |
| ------------- | ------------- | ------------- | ------------- |
| GPT-4o | 90.9 | 97.74 | + 6.84 |
| Claude 3.5 Sonnet v2 | 88.58 | 96.49 | +7.91 |
| Gemini 1.5-pro-002 | 88.17 | 94.19 | +6.02 |
Overall, the prompt engineering somewhat significantly improved the
generation efficiency.
## Summary
Turns out, claude can in some situations (when returning both text and
toolcall in a single message) starts their toolcall index at `1` instead
of `0`, which introducing null values in the concatenated messages.
This fixes it, by removing null values from the tool calls when merging
the chunks.
Also remove the SKA codeowner override for the inference plugin to get
back the shared ownership
## Summary
Adjusts navigating to pages. Directly navigates to some pages instead of
clicking on a button to navigate. Also adds video for certain tests to
help with debugging any flakiness.
## Summary
* Delete unused package `@kbn/formatters`
* Delete unused package `@kbn/response-ops-feature-flag-service`
* Flag `@kbn/generate-console-definitions` as `devOnly: true`
* Flag `@kbn/plugin-check` as `devOnly: true`
* Flag `@kbn/set-map` as `devOnly: true`
* Flag `@kbn/synthetics-private-location` as `devOnly: true`
* Categorise `@kbn/calculate-auto` as `platform/shared`
* Categorise `@kbn/charts-theme` as `platform/shared`
* Categorise `@kbn/saved-search-component` as `platform/shared`
* Categorise `@kbn/use-tracked-promise` as `platform/shared`
* Categorise `@kbn/charts-theme` as `platform/shared`
* Categorise `@kbn/response-ops-rule-form` as `platform/shared`
## Summary
Fix https://github.com/elastic/kibana/issues/204559
Use the default ELSER deployment (`.elser-2-elasticsearch`) for the
product documentation semantic_text fields instead of maintaining our
own custom deployment.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Stacked on https://github.com/elastic/kibana/pull/204004
<img width="1275" alt="Screenshot 2024-12-12 at 17 19 58"
src="https://github.com/user-attachments/assets/2ad14305-15c0-4522-8e70-5691c50e381b"
/>
Adds some bits to the stream overview page:
* Number of docs for the current time range (let's stop here and don't
build more of Kibana)
* List of child streams for wired streams
* Quick links tab (currently empty)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Removes `react-syntax-highlighter` from APM errors, in favour of
`EuiCodeBlock` for read-only code syntax highlighting. This in turn
removes a bunch of custom styling to bring things more inline with the
design system as well.
Closes#204049
## How to test
* Go to Applications - Service Inventory
* Find a service with errors
* Go to Errors tab for service
* Select an error that is an exception
* View details for the exception and see the syntax highlighted block
for the stack trace.
Closes: #205544
## Description
When user tabs over sync alert status with case status toggle button
under case settings on create case page, screenreader announces On, On
switch without giving any context.
## Preconditions
Security solution -> on cases page -> create case
## Changes made:
1. added context for **EuiSwitch** by passing `aria-labelledby`
attribute
## Screen
[Security Solution] [Attack discovery] Update Attack Discovery
evaluation prompts
This PR updates prompts used to evaluate the initial outputs of Attack
Discovery.
Only text was changed.
## Summary
This PR fixes the performance test pipelines by removing the bits that
rely on the plugins build.
### Details
* The plugin build has been removed with #197125. Since the performance
pipelines are running against a Kibana build (and not against sources),
they should not need the plugin build.
* The `performance-data-set-extraction` pipeline started to fail
immediately after the plugin build has been removed
* This failure went unnoticed since the `scalability-benchmarking`
pipeline continued to work by using the last uploaded artifacts from the
`performance-data-set-extraction` pipeline, which were available for
another month. Once the old artifacts were no longer available, the
`scalability-benchmarking` pipeline also started to fail.
## Summary
This PR removes the `analyzerDatePickersAndSourcererDisabled` feature
flag that was introduced a long time ago and has been in `disabled:
false` state for many months.
I noticed that the line was moved in [this
PR](https://github.com/elastic/kibana/pull/176064) over 6 months ago but
the introduction of the feature precedes that.
No UI changes introduced!
## Summary
[Internal link](https://github.com/elastic/security-team/issues/10820)
to the feature details
These changes add a functionality which enables related integrations
functionality for migration rules:
* related integration are shown in the migration rules table
* user can navigate to the integration page to see instructions about
installation process
### Other tasks and fixes
* Default sorting in the table (by `Stats` => by `Author` => by
`Severity` => by `Updated`)
> [!NOTE]
> This feature needs `siemMigrationsEnabled` experimental flag enabled
to work.
## Screen recording
<img width="1838" alt="Screenshot 2024-12-17 at 19 26 47"
src="https://github.com/user-attachments/assets/c1ed9d5d-e237-4dfe-b144-a80adbf46cd3"
/>
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR aims at relocating some of the Kibana modules (plugins and
packages) into a new folder structure, according to the _Sustainable
Kibana Architecture_ initiative.
> [!IMPORTANT]
> * We kindly ask you to:
> * Manually fix the errors in the error section below (if there are
any).
> * Search for the `packages[\/\\]` and `plugins[\/\\]` patterns in the
source code (Babel and Eslint config files), and update them
appropriately.
> * Manually review
`.buildkite/scripts/pipelines/pull_request/pipeline.ts` to ensure that
any CI pipeline customizations continue to be correctly applied after
the changed path names
> * Review all of the updated files, specially the `.ts` and `.js` files
listed in the sections below, as some of them contain relative paths
that have been updated.
> * Think of potential impact of the move, including tooling and
configuration files that can be pointing to the relocated modules. E.g.:
> * customised eslint rules
> * docs pointing to source code
> [!NOTE]
> * This PR has been auto-generated.
> * Any manual contributions will be lost if the 'relocate' script is
re-run.
> * Try to obtain the missing reviews / approvals before applying manual
fixes, and/or keep your changes in a .patch / git stash.
> * Please use
[#sustainable_kibana_architecture](https://elastic.slack.com/archives/C07TCKTA22E)
Slack channel for feedback.
Are you trying to rebase this PR to solve merge conflicts? Please follow
the steps describe
[here](https://elastic.slack.com/archives/C07TCKTA22E/p1734019532879269?thread_ts=1734019339.935419&cid=C07TCKTA22E).
#### 3 packages(s) are going to be relocated:
| Id | Target folder |
| -- | ------------- |
| `@kbn/code-editor` |
`src/platform/packages/shared/shared-ux/code_editor/impl` |
| `@kbn/code-editor-mock` |
`src/platform/packages/shared/shared-ux/code_editor/mocks` |
| `@kbn/monaco` | `src/platform/packages/shared/kbn-monaco` |
<details >
<summary>Updated relative paths</summary>
```
src/platform/packages/shared/kbn-monaco/jest.config.js:12
src/platform/packages/shared/kbn-monaco/tsconfig.json:2
src/platform/packages/shared/kbn-monaco/tsconfig.type_check.json:2
src/platform/packages/shared/shared-ux/code_editor/impl/jest.config.js:12
src/platform/packages/shared/shared-ux/code_editor/impl/tsconfig.json:16
src/platform/packages/shared/shared-ux/code_editor/impl/tsconfig.json:2
src/platform/packages/shared/shared-ux/code_editor/impl/tsconfig.type_check.json:18
src/platform/packages/shared/shared-ux/code_editor/impl/tsconfig.type_check.json:2
src/platform/packages/shared/shared-ux/code_editor/impl/tsconfig.type_check.json:25
src/platform/packages/shared/shared-ux/code_editor/impl/tsconfig.type_check.json:28
src/platform/packages/shared/shared-ux/code_editor/impl/tsconfig.type_check.json:31
src/platform/packages/shared/shared-ux/code_editor/impl/tsconfig.type_check.json:34
src/platform/packages/shared/shared-ux/code_editor/impl/tsconfig.type_check.json:37
src/platform/packages/shared/shared-ux/code_editor/impl/tsconfig.type_check.json:40
src/platform/packages/shared/shared-ux/code_editor/mocks/tsconfig.json:16
src/platform/packages/shared/shared-ux/code_editor/mocks/tsconfig.json:2
src/platform/packages/shared/shared-ux/code_editor/mocks/tsconfig.type_check.json:18
src/platform/packages/shared/shared-ux/code_editor/mocks/tsconfig.type_check.json:2
src/platform/packages/shared/shared-ux/code_editor/mocks/tsconfig.type_check.json:25
```
</details>
closes https://github.com/elastic/observability-dev/issues/3777
## Summary
This PR provides scale dimensions for the service map and infra host
pages without introducing any additional requests.
### Global Service Map and Per-service APM Service Map
| Metric | Description |
|----------------|------------------------------------|
| `num_of_nodes` | Total number of discovered nodes (services +
dependenies |
| `num_of_traces` | Total number of traces |
### Infra
| Metric | Description | default
|---------------|---------------------------------| -----------------
| `num_of_hosts` | Total number of hosts |
| `max_hosts_per_page` | Maximum number of host returne `50/100/500` |
100
| Page | Screenshot |
|-------------------------------|-------------------------------------------------------------------------------------------|
| Global Service Map | 
|
| Per-service APM Service Map | 
|
| Infra | !
|
### How to test
- Open any of the above pages
- In the network tab, look for `kibana:plugin_render_time`
## Summary
Closes https://github.com/elastic/kibana/issues/202295
Closes https://github.com/elastic/kibana/issues/202296
This PR adapts Inventory to use the new Entity v2 endpoints.
## Testing
- Use any synthtrace scenario that loads service/hosts/containers data
- Navigate and make sure everything works as expected (navigation to
Discovery/Infra/Services pages, interacting with the table, searching
for some specific entity, interacting with the type filter)
- To check the alerts work, it's easier to connect to a remote cluster.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Jenny <dzheni.pavlova@elastic.co>
This PR updates the console definitions to match the latest ones from
the @elastic/elasticsearch-specification repo.
Co-authored-by: Elena Stoeva <59341489+ElenaStoeva@users.noreply.github.com>
**Resolves: #200167**
## Summary
Increase number of fetched package policies to the maximum. Currently
only the first 20 policies (the first page) are returned, which results
in treating all remaining ones as disabled.
I am proposing the simplest change of increasing the limit here to the
maximum. There shouldn't be too many policies there, e.g. in the
reproduction I am running there are 23 instead of 20.
If that is not enough, however, the alternative would be to discover
that there are more policies than the specified limit and the next
page(s) would have to be collected and the results added to the final
list.
#BEFORE
#AFTER
### Checklist
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
This closes https://github.com/elastic/streams-program/issues/54.
The root stream is selectively immutable (processing and fields changes
are not allowed).
## UI
For the UI I've entirely disabled the actions column for the root stream
in the schema editor. All of the information (bar the preview table for
changes) available in the flyout for a field is already available in the
table, so this seems easiest for now to avoid multiple logic forks
wrapping buttons etc.
E.g. flyout vs table
## Summary
[Internal link](https://github.com/elastic/security-team/issues/10820)
to the feature details
These changes add a functionality which allows user to retry failed
migration rules.
### Other tasks and fixes
* Integrated `MigrationReadyPanel` and `MigrationProgressPanel` to show
migration's `ready` and `running` states
* Migration stats pooling issue caused by waiting while there are no
pending migrations left. If any other operation triggers `startPooling`
during the waiting it will be ignored and thus latest stats will never
come back.
> [!NOTE]
> This feature needs `siemMigrationsEnabled` experimental flag enabled
to work.
### Testing note
1. Make sure you have a SIEM migration with failed rules
2. Open that migration via `Security > Rules > SIEM Rules Migrations >
{#MIGRATION_WITH_FAILED_RULES}`
3. You should see a `Reprocess rules (#)` button which triggers failed
rules reprocessing
## Screen recording
https://github.com/user-attachments/assets/d33dc4a0-1791-4869-aa8d-b0322b5f19c3
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
**Resolves:** https://github.com/elastic/kibana/issues/200134
## Summary
This PR implements concurrency control to make sure user has the recent rule updates data in Rule Upgrade flyout. Any modifications saved in Rule Upgrade flyout are reset upon new `revision` or `version` detected.
## Details
Concurrency control is important to provide better UX. Multiple users work in Kibana in parallel and new prebuilt rules package version can be released in any time. Attempts to upgrade a rule with outdated `revision` and/or `version` results in failed request. Users may experience multiple rule upgrade failure in that case causing a lot of confusion. More experienced users may guess to reload the page to continue.
Typical reasons leading to `revision` and/or `version` change are the following
- Current rule has been edited will bump rule's `revision`. For example the rule currently shown in Rule Upgrade flyout has been edited by someone else.
- Prebuilt rules package got released will give provide rule assets with higher `version`. Rules having upgrades in the currently installed package and in a new one are affected.
This PR mitigates the described issues by implementing concurrency control. It sets up `_review` API endpoint refetch interval to 5 minutes to fetch fresh data. In case a higher `revision` or `version` is detected for some rule this rule's resolved conflicts and customizations performed in Rule Upgrade flyout get cleared.
## Screenshots
- `revision` change (refresh interval was reduced to 30 seconds to make the video shorter)
https://github.com/user-attachments/assets/98d2a22f-9338-482a-a7b2-1e170b9642ce
- `version` change (refresh interval was reduced to 1 minute to make the video shorter)
https://github.com/user-attachments/assets/2b7c23f0-5a50-471e-aa7f-8d9b2aecc957
## How to test locally
There are two cases for testing
- `revision` change
- `version` change
### Test `revision` change
Revision change means the rule has been edited. Use the following steps to test it
- Ensure the `prebuiltRulesCustomizationEnabled` feature flag is enabled
- Allow internal APIs via adding `server.restrictInternalApis: false` to `kibana.dev.yaml`
- Clear Elasticsearch data
- Run Elasticsearch and Kibana locally (do not open Kibana in a web browser)
- Install an outdated version of the `security_detection_engine` Fleet package
```bash
curl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 2023-10-31" -d '{"force":true}' http://localhost:5601/kbn/api/fleet/epm/packages/security_detection_engine/8.14.1
```
- Install prebuilt rules
```bash
curl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 1" -d '{"mode":"ALL_RULES"}' http://localhost:5601/kbn/internal/detection_engine/prebuilt_rules/installation/_perform
```
- Open `Detection Rules (SIEM)` Page -> `Rule Updates`
- Open Rule upgrade flyout for some rule
- Make changes to rule field(s) and save them (do not upgrade the rule)
- Open the other web browser tab with Kibana
- Navigate to the same rule's editing page
- Change any field and save the changes
- Return back to the first tab and wait for data to be refetched (data refresh interval is 5 minutes, wait for `_review` request in the Dev Tool's Network tab)
- Make sure the changes you made for field(s) got reverted
### Test `version` change
Version change means a new package version was released. Do the following to test it
- Ensure the `prebuiltRulesCustomizationEnabled` feature flag is enabled
- Allow internal APIs via adding `server.restrictInternalApis: false` to `kibana.dev.yaml`
- Clear Elasticsearch data
- Run Elasticsearch and Kibana locally (do not open Kibana in a web browser)
- Set `xpack.securitySolution.prebuiltRulesPackageVersion: 8.15.2` in `kibana.dev.yaml`
- Install an outdated version of the `security_detection_engine` Fleet package
```bash
curl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 2023-10-31" -d '{"force":true}' http://localhost:5601/kbn/api/fleet/epm/packages/security_detection_engine/8.14.1
```
- Install prebuilt rules
```bash
curl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 1" -d '{"mode":"ALL_RULES"}' http://localhost:5601/kbn/internal/detection_engine/prebuilt_rules/installation/_perform
```
- Open `Detection Rules (SIEM)` Page -> `Rule Updates`
- Open Rule upgrade flyout for a rule having updates in packages `v8.15.2` and `.8.17.1-beta.1` for example `Suspicious Web Browser Sensitive File Access`
- Make changes to rule field(s) and save them (do not upgrade the rule)
- Set `xpack.securitySolution.prebuiltRulesPackageVersion: 8.17.1-beta.1` in `kibana.dev.yaml`
- Open the other web browser tab with Kibana
- Navigate to Security Solution plugin to install the
OR
install the package `8.17.1-beta.1` via API request
```bash
curl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 2023-10-31" -d '{"force":true}' http://localhost:5601/kbn/api/fleet/epm/packages/security_detection_engine/8.17.1-beta.1
```
- Return back to the first tab and wait for data to be refetched (data refresh interval is 5 minutes, wait for `_review` request in the Dev Tool's Network tab)
- Make sure the changes you made for field(s) got the recent target rule values
Alternatively you can spin up EPR locally and publish package updates with rule's version bumped.
