Closes https://github.com/elastic/kibana/issues/190795
Adds the logic to display a message to the user in case there is already
an existing Firehose data in their cluster and to show the identified
AWS services in the "Visualize Data" step right away without waiting for
the window to loose focus first.
## How to test
1. Open the Firehose flow
2. Make sure there is no callout and the third step is not active
3. Go to Kibana dev console and ingest some dummy data (see examples
bellow)
4. Refresh the page with the Firehose flow
5. make sure there is a callout and the third steps shows the identified
AWS service
```
POST logs-aws.apigateway_logs-default/_doc
{
"@timestamp": "2024-11-25T13:32:01.000Z",
"some": 111,
"aws.kinesis.name": "Elastic-CloudwatchLogs"
}
POST metrics-aws.apigateway_metrics-default/_doc
{
"@timestamp": "2024-11-25T13:31:01.000Z",
"agent": {
"type": "firehose"
},
"aws": {
"cloudwatch": {
"namespace": "AWS/ApiGateway"
},
"exporter": {
"arn": "arn:aws:cloudwatch:us-west-2:975050175126:metric-stream/Elastic-CloudwatchLogsAndMetricsToFirehose-CloudWatchMetricStream-Nhb4NhzPdL4J"
}
},
"cloud": {
"account": {
"id": "975050175126"
},
"provider": "aws",
"region": "us-west-2"
}
}
```
## Summary
This PR aims at relocating some of the Kibana modules (plugins and
packages) into a new folder structure, according to the _Sustainable
Kibana Architecture_ initiative.
> [!IMPORTANT]
> * We kindly ask you to:
> * Manually fix the errors in the error section below (if there are
any).
> * Search for the `packages[\/\\]` and `plugins[\/\\]` patterns in the
source code (Babel and Eslint config files), and update them
appropriately.
> * Manually review
`.buildkite/scripts/pipelines/pull_request/pipeline.ts` to ensure that
any CI pipeline customizations continue to be correctly applied after
the changed path names
> * Review all of the updated files, specially the `.ts` and `.js` files
listed in the sections below, as some of them contain relative paths
that have been updated.
> * Think of potential impact of the move, including tooling and
configuration files that can be pointing to the relocated modules. E.g.:
> * customised eslint rules
> * docs pointing to source code
> [!NOTE]
> This PR has been auto-generated.
> Do not attempt to push any changes unless you know what you are doing.
> Please use
[#sustainable_kibana_architecture](https://elastic.slack.com/archives/C07TCKTA22E)
Slack channel for feedback.
#### 1 plugin(s) are going to be relocated:
| Id | Target folder |
| -- | ------------- |
| `@kbn/translations-plugin` |
`x-pack/platform/plugins/private/translations` |
<details>
<summary>Updated references</summary>
```
./docs/developer/plugin-list.asciidoc
./package.json
./packages/kbn-repo-packages/package-map.json
./packages/kbn-synthetic-package-map/synthetic-packages.json
./packages/kbn-ts-projects/config-paths.json
./src/dev/i18n_tools/README.md
./tsconfig.base.json
./tsconfig.refs.json
./yarn.lock
```
</details>
<details>
<summary>Updated relative paths</summary>
```
x-pack/platform/plugins/private/translations/tsconfig.json:2
```
</details>
<details>
<summary>Script errors</summary>
```
```
</details>
## Summary
Closes https://github.com/elastic/ingest-dev/issues/4325
Added modal window with column selector table when agents are exported.
There are some differences compared to the design due to technical
limitations:
- `Filters applied` badge not included, it's not really possible to
accurately calculate the count of filters, since they are stored in a
single string. We could potentially count by splitting AND/OR
conditions, but it may not be accurate. Do we still want to include it?
- We don't have display names for these mappings coming from the agent
index, and currently I don't have a way to show display names on the
exported file. For this reason showing the original field names on the
modal too.
- Added a Description column that transforms the field name to a more
readable name, we can also hardcode the descriptions if this is not good
enough.
- After some consideration, I decided to hardcode the allowed field list
to export other than querying dynamically from the agent index mappings,
otherwise new sensitive field mappings would show up on the UI.
- Caveat: searching on columns in the modal removes the already selected
columns that do not match. This seems to be a default behaviour of the
EuiTable, I find it a little strange, didn't see a straightforward way
to change it. I can spend more time on it if needed.
To verify:
- Select a few agents, click on Export CSV action
- Verify that the modal window opens with the agent mappings visible, by
default the columns on the UI selected
- Select a few columns to export
- Submit the modal
- Wait for the report to be ready and download it
- Verify that the exported csv includes the columns selected
<img width="1301" alt="image"
src="https://github.com/user-attachments/assets/71b549a7-a316-45d4-b017-f5eedec6ea7e">
Figma design:
<img width="2138" alt="image"
src="https://github.com/user-attachments/assets/77ffc5d7-343f-4d5c-8b65-251e1cac94d4">
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
With https://github.com/elastic/elasticsearch/issues/117544 fixed,
`subobjects: false` can be set on wired streams.
This allows to map fields as scalars even if they share the same prefix,
e.g.
* `deeply.nested.path` as `keyword`
* `deeply.nested.path.and.more` as `keyword`
Closes https://github.com/elastic/kibana/issues/203440
### Summary
Update session cleanup task by adding the partial search results flag to
the PIT query as well and not just the search query.
#### Notes
In the previous “fix”, the partial search results flag was incorrectly
added to the search query that depended on the PIT query. However, the
correct way is to set the flag when we openPointInTimeQuery which is
then used in the subsequent search query
### Release notes
Fixes error with opening point in time query for session deletion by now
accounting for partial results.
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_node:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Summary
This PR aims at relocating some of the Kibana modules (plugins and
packages) into a new folder structure, according to the _Sustainable
Kibana Architecture_ initiative.
> [!IMPORTANT]
> * We kindly ask you to:
> * Manually fix the errors in the error section below (if there are
any).
> * Search for the `packages[\/\\]` and `plugins[\/\\]` patterns in the
source code (Babel and Eslint config files), and update them
appropriately.
> * Manually review
`.buildkite/scripts/pipelines/pull_request/pipeline.ts` to ensure that
any CI pipeline customizations continue to be correctly applied after
the changed path names
> * Review all of the updated files, specially the `.ts` and `.js` files
listed in the sections below, as some of them contain relative paths
that have been updated.
> * Think of potential impact of the move, including tooling and
configuration files that can be pointing to the relocated modules. E.g.:
> * customised eslint rules
> * docs pointing to source code
> [!NOTE]
> This PR has been auto-generated.
> Do not attempt to push any changes unless you know what you are doing.
> Please use
[#sustainable_kibana_architecture](https://elastic.slack.com/archives/C07TCKTA22E)
Slack channel for feedback.
#### 3 plugin(s) are going to be relocated:
| Id | Target folder |
| -- | ------------- |
| `@kbn/inference-plugin` | `x-pack/platform/plugins/shared/inference` |
| `@kbn/llm-tasks-plugin` |
`x-pack/platform/plugins/shared/ai_infra/llm_tasks` |
| `@kbn/product-doc-base-plugin` |
`x-pack/platform/plugins/shared/ai_infra/product_doc_base` |
#### 2 package(s) are going to be relocated:
| Id | Target folder |
| -- | ------------- |
| `@kbn/inference-common` |
`x-pack/platform/packages/shared/ai-infra/inference-common` |
| `@kbn/product-doc-common` |
`x-pack/platform/packages/shared/ai-infra/product-doc-common` |
<details>
<summary>Updated references</summary>
```
./docs/developer/plugin-list.asciidoc
./package.json
./packages/kbn-repo-packages/package-map.json
./packages/kbn-ts-projects/config-paths.json
./tsconfig.base.json
./x-pack/.i18nrc.json
./x-pack/platform/packages/shared/ai-infra/inference-common/jest.config.js
./x-pack/platform/packages/shared/ai-infra/product-doc-common/jest.config.js
./x-pack/platform/plugins/shared/ai_infra/llm_tasks/jest.config.js
./x-pack/platform/plugins/shared/ai_infra/product_doc_base/jest.config.js
./x-pack/platform/plugins/shared/inference/jest.config.js
./x-pack/platform/plugins/shared/inference/scripts/evaluation/README.md
./yarn.lock
```
</details>
<details>
<summary>Updated relative paths</summary>
```
x-pack/platform/packages/shared/ai-infra/inference-common/jest.config.js:10
x-pack/platform/packages/shared/ai-infra/inference-common/tsconfig.json:2
x-pack/platform/packages/shared/ai-infra/product-doc-common/jest.config.js:10
x-pack/platform/packages/shared/ai-infra/product-doc-common/tsconfig.json:2
x-pack/platform/plugins/shared/ai_infra/llm_tasks/jest.config.js:10
x-pack/platform/plugins/shared/ai_infra/llm_tasks/tsconfig.json:2
x-pack/platform/plugins/shared/ai_infra/llm_tasks/tsconfig.json:7
x-pack/platform/plugins/shared/ai_infra/product_doc_base/jest.config.js:10
x-pack/platform/plugins/shared/ai_infra/product_doc_base/tsconfig.json:2
x-pack/platform/plugins/shared/ai_infra/product_doc_base/tsconfig.json:7
x-pack/platform/plugins/shared/inference/jest.config.js:10
x-pack/platform/plugins/shared/inference/scripts/util/read_kibana_config.ts:16
x-pack/platform/plugins/shared/inference/tsconfig.json:2
x-pack/platform/plugins/shared/inference/tsconfig.json:7
```
</details>
<details>
<summary>Script errors</summary>
```
```
</details>
---------
Co-authored-by: pgayvallet <pierre.gayvallet@elastic.co>
I noticed that a previous PR broke the functionality of creating wired
streams implicitly by mentioning them as children of the parent stream
in the `PUT /api/streams/<stream name>` API call because the check for
classic streams would throw when trying to retrieve a non-existing data
stream in
```
await scopedClusterClient.asInternalUser.indices.getDataStream({ name: id });
```
This PR fixes the problem and introduces a new integration test for the
use case of "flushing" a list of streams one after the other, as this
will probably be a common thing for users automating the API.
**Resolves:** https://github.com/elastic/kibana/issues/203365
## Summary
This PR fixes a bug preventing Threat Match rules from being upgraded.
## Details
It's expected that users can upgrade Elastic prebuilt rules without any customizations in any possible way (in flyout, rules upgrade table, bulk actions). This operation was blocked due to wrong Threat Match fields conversions to Diffable rule. This PR fixes rule type specific fiels conversions and merged `threat_query` field with `threat_query`.
## Screen recording (with the fix)
https://github.com/user-attachments/assets/9f0375fb-d39d-4b4b-a084-96ea265f306f
Closes https://github.com/elastic/kibana/issues/202255
### Summary
While working on the visual refresh for the new EUI theme Borealis we
figured that was a good time to do the recommended migration from
styled-components to @emotion
### What has been done
- Migrate infra plugin from styled-components to @emotion
- Eui Visual Refresh for Borealis new theme
- All usage of color palette tokens and functions now pull from the
theme, and correctly update to use new colors when the theme changes
from Borealis to Amsterdam and vice versa
- All references to renamed tokens have been updated to use the new
token name
- Remove usage of deprecated `useEuiBackgroundColor`
- All usages of "success" colors have been updated to `accentSecondary`
and `textAccentSecondary` as needed
### How to test
#### Running Kibana with the Borealis theme
In order to run Kibana with Borealis, you'll need to do the following:
- Set the following in kibana.dev.yml:
`uiSettings.experimental.themeSwitcherEnabled: true`
- Run Kibana with the following environment variable set:
`KBN_OPTIMIZER_THEMES="borealislight,borealisdark,v8light,v8dark" yarn
start`
- This will expose a toggle under Stack Management > Advanced Settings >
Theme version, which you can use to toggle between Amsterdam and
Borealis.
## Summary
**Reviewers: Please test the code paths affected by this PR. See the
"Risks" section below.**
Part of work for enabling "high contrast mode" in Kibana. See
https://github.com/elastic/kibana/issues/176219.
**Background:**
Kibana will soon have a user profile setting to allow users to enable
"high contrast mode." This setting will activate a flag with
`<EuiProvider>` that causes EUI components to render with higher
contrast visual elements. Consumer plugins and packages need to be
updated selected places where `<EuiProvider>` is wrapped, to pass the
`UserProfileService` service dependency from the CoreStart contract.
**NOTE:** **EUI currently does not yet support the high-contrast mode
flag**, but support for that is expected to come in around 2 weeks.
These first PRs are simply preparing the code by wiring up the
`UserProvideService`.
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [X] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [X] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Risks
Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.
Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.
- [ ] [medium/high] The implementor of this change did not manually test
the affected code paths and relied on type-checking and functional tests
to drive the changes. Code owners for this PR need to manually test the
affected code paths.
- [ ] [medium] The `UserProfileService` dependency comes from the
CoreStart contract. If acquiring the service causes synchronous code to
become asynchronous, check for race conditions or errors in rendering
React components. Code owners for this PR need to manually test the
affected code paths.
## Summary
[Internal link](https://github.com/elastic/security-team/issues/10820)
to the feature details
These changes add sorting functionality to the migration rules table. It
is possible to sort migration rules by next columns: `Updated`, `Name`,
`Status`, `Risk Score`, `Severity` and `Author`.
### Other changes
Next fixes and adjustments were also implemented as part of this PR:
* `Installed` status in migration rules table to indicate whether the
rule was installed
* Rules selection and installation of selected rules
* Disable selection for not fully translated rules
* `Author` column to show whether the translated rule matched one of the
existing Elastic prebuilt rules
* `Install and enable` and `Install without enabling` buttons within the
migration rule details flyout
## Summary
This PR removes rendering of the Inference Endpoints UI from
`enterprise_search` for stack and instead utilizes the
`search_inference_endpoints` plugin to render the UI for both serverless
and stack. This can be done by utilizing the `search_navigation` plugin
for rendering the classic navigation for stack.
To support this change the `xpack.searchInferenceEndpoints.ui.enabled`
was updated to default to `true` instead of only being set for
serverless search. To account for this change I have added
`xpack.searchInferenceEndpoints.enabled: false` to the serverless
configs for both security and observability to ensure the
`search_inference_endpoints` plugin is disabled in both of those
projects.
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
### Problem
Cookie authentication was introduced in Kibana for serverless internal
API tests via https://github.com/elastic/kibana/pull/192727.
The serverless tests for Obs AI Assistant still uses API key based auth.
### Solution
Change authentication to cookie based auth for internal APIs in
serverless tests.
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Summary
### Service Definition:
https://github.com/elastic/kibana/pull/202344/files#diff-42c7dd345e0500c97f85824904a70a11162827ea8f8df6982082a9047ca04ff1
### Acceptance Criteria
- [x] Upon installation of the entity store, the Service entity
definition should be created by default
- [x] The Service definition will be installed in the exact same way as
the User and Host definitions
- [x] The unique identifier for service entities will be `service.name`
- [x] The fields captured for service entities should match the field
mapping spreadsheet (see Implementation Notes below)
### Stored Entity
```json
{
"@timestamp": "2024-12-02T10:43:13.856Z",
"event": {
"ingested": "2024-12-02T10:51:28.987428Z"
},
"entity": {
"name": "test123 name",
"id": "test123 name",
"source": "logs-blito",
"type": "service"
},
"service": {
"node": {
"roles": [
"test123 node roles"
],
"name": [
"test123 node name"
]
},
"environment": [
"test123 environment"
],
"address": [
"test123 address"
],
"name": "test123 name",
"id": [
"test123 id"
],
"state": [
"test123 state"
],
"ephemeral_id": [
"test123 ephemeral_id"
],
"type": [
"test123 type"
],
"version": [
"test123 version"
]
}
}
```
### How to test it?
* Start Kibana
<details>
<summary>Create mappings</summary>
```
PUT /logs-test
{
"mappings": {
"properties": {
"service.name": {
"type": "keyword"
},
"service.address": {
"type": "keyword"
},
"service.environment": {
"type": "keyword"
},
"service.ephemeral_id": {
"type": "keyword"
},
"service.id": {
"type": "keyword"
},
"service.node.name": {
"type": "keyword"
},
"service.node.roles": {
"type": "keyword"
},
"service.state": {
"type": "keyword"
},
"service.type": {
"type": "keyword"
},
"service.version": {
"type": "keyword"
},
"@timestamp": {
"type": "date"
}
}
}
}
````
</details>
<details>
<summary>Create document</summary>
```
PUT /logs-test
POST logs-test/_doc
{
"service": {
"name": "test123 name",
"address": "test123 address",
"environment": "test123 environment",
"ephemeral_id": "test123 ephemeral_id",
"id": "test123 id",
"node.roles": "test123 node roles",
"node.name": "test123 node name",
"state": "test123 state",
"type": "test123 type",
"version": "test123 version"
},
"@timestamp": "2024-12-02T10:43:13.856Z"
}
````
</details>
* Init the entity store
* Wait...
* Query the service index `GET
.entities.v1.latest.security_service_default/_search`
### Open Questions
* Can we merge this PR without first updating all other features that
will use service entities?
* If we merge it, the service engine will be installed together with
other entities, but it won't provide any functionality
* Do we need an experimental flag?
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR swaps the `EuiSelectable` by the `EuiComboBox` component as we
have in ESS.
### Context:
This PR belongs to this initiative
https://github.com/elastic/search-team/issues/8000 where we agreed on
bring low hanging fruit artefacts from ESS to ES3 before being replace
completly with the full new experience which will be later. Therefor we
are not investing effort on make the code scalable and reusable at this
point.
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Navarone Feekery <13634519+navarone-feekery@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Rodney Norris <rodney@tattdcodemonkey.com>
## Summary
This PR adds logging for the final step of report execution: waiting for
the writable stream's pending callbacks.
* **Debug**: log the beginning of the wait
* **Info**: log the end of the wait
Update: this also puts the report job ID in the context of the logger
utility used for running the task.
<img width="1983" alt="image"
src="https://github.com/user-attachments/assets/1ef68758-c2c9-4b6d-8090-50e496089140">
### Checklist
none
### Identify risks
none
## Summary
* Removes `delete` permission check on `.tasks`
* Task doc deletion best effort
* Creates a `warning` level log if deletion from `.tasks` fails like:
```
[2024-12-09T10:50:28.398+01:00][WARN ][plugins.upgradeAssistant.reindex_worker] ResponseError: security_exception
log.ts:66
Root causes:
log.ts:66
security_exception: action [indices:data/write/bulk[s]] is unauthorized for API key id [___] of user [elastic] on restricted indices [.tasks], this action is granted by the index privileges [create_doc,create,delete,index,write,all]
```
## How to test
1. Follow [these
steps](https://github.com/elastic/kibana-team/issues/1249#issuecomment-2514462816),
but instead of creating a data stream create an index in 7.x
2. Start ES on v8.x
3. Checkout Kibana `8.x` locally and apply [the
diff](https://patch-diff.githubusercontent.com/raw/elastic/kibana/pull/203379.diff)
from this branch
4. Start Kibana
5. Log in as `elastic` or some other admin/superuser
6. Go to UA and reindex the index you created
## Resources
### Outdated 7.x guidance
https://www.elastic.co/guide/en/elasticsearch/reference/7.17/docs-update-by-query.html#docs-update-by-query-task-api
> When you are done with a task, you should delete the task document so
Elasticsearch can reclaim the space.
This PR migrates test suites that use `renderHook` from the library
`@testing-library/react-hooks` to adopt the equivalent and replacement
of `renderHook` from the export that is now available from
`@testing-library/react`. This work is required for the planned
migration to react18.
## Context
In this PR, usages of `waitForNextUpdate` that previously could have
been destructured from `renderHook` are now been replaced with `waitFor`
exported from `@testing-library/react`, furthermore `waitFor`
that would also have been destructured from the same renderHook result
is now been replaced with `waitFor` from the export of
`@testing-library/react`.
***Why is `waitFor` a sufficient enough replacement for
`waitForNextUpdate`, and better for testing values subject to async
computations?***
WaitFor will retry the provided callback if an error is returned, till
the configured timeout elapses. By default the retry interval is `50ms`
with a timeout value of `1000ms` that
effectively translates to at least 20 retries for assertions placed
within waitFor. See
https://testing-library.com/docs/dom-testing-library/api-async/#waitfor
for more information.
This however means that for person's writing tests, said person has to
be explicit about expectations that describe the internal state of the
hook being tested.
This implies checking for instance when a react query hook is being
rendered, there's an assertion that said hook isn't loading anymore.
In this PR you'd notice that this pattern has been adopted, with most
existing assertions following an invocation of `waitForNextUpdate` being
placed within a `waitFor`
invocation. In some cases the replacement is simply a `waitFor(() => new
Promise((resolve) => resolve(null)))` (many thanks to @kapral18, for
point out exactly why this works),
where this suffices the assertions that follow aren't placed within a
waitFor so this PR doesn't get larger than it needs to be.
It's also worth pointing out this PR might also contain changes to test
and application code to improve said existing test.
### What to do next?
1. Review the changes in this PR.
2. If you think the changes are correct, approve the PR.
## Any questions?
If you have any questions or need help with this PR, please leave
comments in this PR.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Graph changes:
This PR focuses on adding the functionality to retrieve currrently
available prebuilt rules and create a new index with semantic_text
mappings to allow the SIEM migration process to use it for RAG usecases.
The below changes are some specific mentions that the PR changes:
- Move the creation of the RAG indicies from `/create` to `/start`, also
removes the `await` for `prepare` when `/start` is called.
- Move all retrievers to a new `retriever` folder, together with a new
`RuleMigrationsRetriever` class to encapsulate all the different
retrievers at one place.
- Adds timeout to integration and prebuilt rule bulk requests to ES
because of the possible time it can take to generate initial embeddings.
- Move some nodes from Translate Rule subgraph to the main agent graph,
as semantic queries are used now for both translate and matching
prebuilt.
## Summary
Closes https://github.com/elastic/kibana/issues/183245Closes#197630
[Observability AI Assistant] Partially migrate to inference client
replacing `inferenceClient.chatComplete` to
`observabilityAIAssistantClient.chat` -
`observabilityAIAssistantClient.complete` does a bunch of stuff on top
of `chat`. keepping `observabilityAIAssistantClient.chat` as a wrapper
for now because it also adds instrumentation and logging.
## Release note
Use Data stream name for data_stream.dataset value in input manifests
## Summary
Closes https://github.com/elastic/kibana/issues/201478
This PR uses the user-supplied data stream name as the
`data_stream.dataset` value instead of a generic placeholder.
Custom Fleet integrations default to a generic dataset
(data_stream.dataset) value - something like `http_endpoint.generic.`
Since users configure their own options for a custom integration, this
workflow makes sense.
Elastic integrations, on the other hand, default to a predetermined
naming convention based on the integration package + datastream name for
dataset - something like github.issues for the issues data stream from
the Github package.
Hence defaulting it to `package_name.datastream_name`
<img width="729" alt="Screenshot 2024-12-05 at 15 12 32"
src="https://github.com/user-attachments/assets/2dad5de3-a329-403f-be46-2dbc76906b03">
<img width="868" alt="Screenshot 2024-12-05 at 15 12 16"
src="https://github.com/user-attachments/assets/2959e26a-8095-4efa-bf9b-c5e953e849b9">
### Checklist
Check the PR satisfies following conditions.
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
Closes [#148241](https://github.com/elastic/kibana/issues/148241)
## Summary
This PR introduces a few changes in the snapshot and restore plugin.
#### Snapshots tab
* The state column has been aded to the table
* In the detail flyout, it has been changed the way in which the status
is displayed (icon and text).
* No new test needed since this info was already there. The related
tests have been updated.
<table>
<tr>
<td style="padding-right: 10px;">
<img alt="Screenshot 2024-11-27 at 15 17 03"
src="https://github.com/user-attachments/assets/92b5c5cf-d90b-454c-9cf7-2eb101562c5f">
</td>
<td>
<img alt="Screenshot 2024-11-27 at 15 54 32"
src="https://github.com/user-attachments/assets/62395bad-4c70-44bb-84b3-c8a30ae9278a">
</td>
</tr>
</table>
#### Policies tab
* The copy of the callout for warning that two or more policies have the
same schedule has been changed. For testing that, you need to have two
or more policies that have the same time por execution. No test added
for this, is only a copy change
[[code](https://github.com/elastic/kibana/pull/199622/files#diff-e8e12f0dfdc97e4e064f8a07965312c8c91ca66578bdcaf5ee807e879cebcb6eR207)]
<img width="1249" alt="Screenshot 2024-11-27 at 15 17 25"
src="https://github.com/user-attachments/assets/5417910f-573a-4c22-a5e2-44ec2ce256b3">
* A new callout has been added to warn that the SLM status is different
from “running”. SLM status is "running" by default, but it could happen
that the user stop it for any reason and then does not restart it. This
causes policies not to run when they are scheduled. To known the SLM
status a new api call has been introduced (`GET _slm/status`).
* I've created a new doc link to
https://www.elastic.co/guide/en/elasticsearch/reference/current/slm-api-start.html
* I've added new tests for the new api call.
<img width="1246" alt="Screenshot 2024-11-27 at 15 59 56"
src="https://github.com/user-attachments/assets/cd5ae491-16a4-4a68-a223-f44ce398ac2f">
## Testing
You will need to have at least one policy to test this. For that, you
can run Elastic using the following:
```
yarn es snapshot --license=trial -E path.repo=/tmp/es-backups
```
From the console, you can add a repository (you can also do it from the
UI):
```
PUT /_snapshot/my_backup
{
"type": "fs",
"settings": {
"location": "/tmp/es-backups",
"chunk_size": "10mb"
}
}
````
And for creating a policy you can run this:
```
PUT _slm/policy/nightly-snapshots
{
"schedule": "0 30 1 * * ?",
"name": "<nightly-snap-{now/d}>",
"repository": "my_backup1",
"retention": {
"expire_after": "30d",
"min_count": 5,
"max_count": 50
}
}
```
SLM status should be started by default (unless you have stopped it).
For starting it you can use `POST /_slm/start` and for stop it `POST
/_slm/stop`
## Demo
https://github.com/user-attachments/assets/b83cd3ba-4821-4295-87f2-ecf427ec46e0
---------
Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>
## Summary
[Internal link](https://github.com/elastic/security-team/issues/10820)
to the feature details
These changes add functionality that allows to display matched prebuilt
rules details.
### New route
There is a new route
`/internal/siem_migrations/rules/{migration_id}/prebuilt_rules` that
will return all prebuilt rules matched by translated rules within a
specific migration.
### UI changes
The rule migration details flyout was updated to display matched
prebuilt rule data in both `Translation` and `Overview` tabs.
https://github.com/user-attachments/assets/3da49653-e0ab-4d8b-892e-dd05cf73743b
### Other changes
Also, as part of this PR, batching of a rule installation/creation was
added.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Sergi Massaneda <sergi.massaneda@gmail.com>
## Summary
This PR aims at relocating some of the Kibana modules (plugins and
packages) into a new folder structure, according to the _Sustainable
Kibana Architecture_ initiative.
> [!IMPORTANT]
> * We kindly ask you to:
> * Manually fix the errors in the error section below (if there are
any).
> * Search for the `packages[\/\\]` and `plugins[\/\\]` patterns in the
source code (Babel and Eslint config files), and update them
appropriately.
> * Manually review
`.buildkite/scripts/pipelines/pull_request/pipeline.ts` to ensure that
any CI pipeline customizations continue to be correctly applied after
the changed path names
> * Review all of the updated files, specially the `.ts` and `.js` files
listed in the sections below, as some of them contain relative paths
that have been updated.
> * Think of potential impact of the move, including tooling and
configuration files that can be pointing to the relocated modules. E.g.:
> * customised eslint rules
> * docs pointing to source code
> [!NOTE]
> This PR has been auto-generated.
> Do not attempt to push any changes unless you know what you are doing.
> Please use
[#sustainable_kibana_architecture](https://elastic.slack.com/archives/C07TCKTA22E)
Slack channel for feedback.
#### 3 plugin(s) are going to be relocated:
| Id | Target folder |
| -- | ------------- |
| `@kbn/entities-data-access-plugin` |
`x-pack/solutions/observability/plugins/observability_solution/entities_data_access`
|
| `@kbn/entityManager-app-plugin` |
`x-pack/solutions/observability/plugins/observability_solution/entity_manager_app`
|
| `@kbn/entityManager-plugin` |
`x-pack/platform/plugins/shared/entity_manager` |
#### 1 package(s) are going to be relocated:
| Id | Target folder |
| -- | ------------- |
| `@kbn/entities-schema` |
`x-pack/platform/packages/shared/kbn-entities-schema` |
<details>
<summary>Updated references</summary>
```
./docs/developer/plugin-list.asciidoc
./package.json
./packages/kbn-repo-packages/package-map.json
./packages/kbn-ts-projects/config-paths.json
./tsconfig.base.json
./x-pack/platform/packages/shared/kbn-entities-schema/jest.config.js
./x-pack/platform/plugins/shared/entity_manager/jest.config.js
./x-pack/plugins/entity_manager/docs/entity_definitions.md
./x-pack/solutions/observability/plugins/observability_solution/entities_data_access/jest.config.js
./x-pack/solutions/observability/plugins/observability_solution/entity_manager_app/jest.config.js
./yarn.lock
```
</details>
<details>
<summary>Updated relative paths</summary>
```
x-pack/platform/packages/shared/kbn-entities-schema/jest.config.js:10
x-pack/platform/packages/shared/kbn-entities-schema/scripts/generate.js:8
x-pack/platform/packages/shared/kbn-entities-schema/scripts/generate_oas.js:65
x-pack/platform/packages/shared/kbn-entities-schema/scripts/generate_oas.js:8
x-pack/platform/packages/shared/kbn-entities-schema/scripts/serve_oas_ui.js:8
x-pack/platform/packages/shared/kbn-entities-schema/tsconfig.json:2
x-pack/platform/plugins/shared/entity_manager/jest.config.js:10
x-pack/platform/plugins/shared/entity_manager/tsconfig.json:2
x-pack/platform/plugins/shared/entity_manager/tsconfig.json:7
x-pack/solutions/observability/plugins/observability_solution/entities_data_access/jest.config.js:12
x-pack/solutions/observability/plugins/observability_solution/entities_data_access/tsconfig.json:2
x-pack/solutions/observability/plugins/observability_solution/entity_manager_app/jest.config.js:12
x-pack/solutions/observability/plugins/observability_solution/entity_manager_app/tsconfig.json:2
x-pack/solutions/observability/plugins/observability_solution/entity_manager_app/tsconfig.json:7
```
</details>
<details>
<summary>Script errors</summary>
```
```
</details>
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Rudolf Meijering <skaapgif@gmail.com>
