Download, parse and replay loghub data with Synthtrace, for use in the
Streams project. In summary:
- adds a `@kbn/sample-log-parser` package which parses Loghub sample
data, creates valid parsers for extracting and replacing timestamps,
using the LLM
- add a `sample_logs` scenario which uses the parsed data sets to replay
Loghub data continuously as if it were live data
- refactor some parts of Synthtrace (follow-up work captured in
https://github.com/elastic/kibana/issues/212179)
## Synthtrace changes
- Replace custom Logger object with Kibana-standard ToolingLog
- Report progress and estimated time to completion for long-running jobs
- Simplify scenarioOpts (allow comma-separated key-value pairs instead
of just JSON)
- Simplify client initialization
- When using workers, only bootstrap once (in the main thread)
- Allow workers to gracefully shutdown
- Downgrade some logging levels for less noise
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Resolves#212919
We noticed that setting the header `'Content-Type':
'text/event-stream',` didn't work as the browser's native EventSource
implementation.
```JS
return res.ok({
headers: {
'Content-Type': 'text/event-stream',
'Cache-Control': 'no-cache',
},
body: observableIntoEventSourceStream(events$ as unknown as Observable<ServerSentEvent>, {
signal: abortController.signal,
logger,
}),
});
```
The reason, apparently, is that we need to flush the compressor's buffer
negotiated in the HTTP request.
### How to test it:
Run Kibana with examples `yarn start --no-base-path --run-examples
--http2` and open the SSE example app in Kibana. You should see a clock
updating every second in the UI (the clock is coming from the server).
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/211778 and
https://github.com/elastic/kibana/issues/211871
This PR is a follow-up to issue
[#208693](https://github.com/elastic/kibana/issues/208693) and
incorporates the suggestions from its associated PR
[#210612](https://github.com/elastic/kibana/pull/210612).
>[!NOTE]
> Please note that most of the changes in this PR involve moving files,
renaming `data-test-subj` attributes, updating imports, and not creating
the files from scratch.
**What's happening in this PR?**
- Transitioning from data source profile to document profile.
- Splitting the traces document viewer into separate span and
transaction viewers.
- Enhancing the `useTransaction` hook to abort requests on component
unmount.
As noted in the `TODOs`, a follow-up issue will focus on using for
APM-configured indexes to enable the experience and retrieve the
`transaction.name` for spans.
### How to test
Add this to your kibana.dev.yml:
```
discover.experimental.enabledProfiles:
- observability-traces-span-document-profile
- observability-traces-transaction-document-profile
`````
- Go to Discover page
- Select APM data view
- Check your traces data
- Spans and transactions should show the new overview tabs when opening
the flyout
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Cauê Marcondes <55978943+cauemarcondes@users.noreply.github.com>
Co-authored-by: Carlos Crespo <crespocarlos@users.noreply.github.com>
## Summary
Adds serverless security AI4SOC add-on metering.
- emits a ai4soc metering usage record with the corresponding tier every
~20 minutes
- records are unique to the project + hour
- records are backfilled up to a week
- adds usage record backfill functionality to the shared usage reporting
task
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
Unified Search belongs to presentations team so I'm changing the
codeowners entry. There was a [previous
attempt](https://github.com/elastic/kibana/pull/208381/files) of doing
the same thing, but the auto-commit kicked in and reverted the change
for unified_search plugin, since we forgot to add the entry
in`kibana.jsonc`.
## Summary
This is intended to be a "minimal" migration for Defend Insights to
langgraph + output chunking. Other than the increased events due to the
context increase from output chunking, the functionality is unchanged.
* migrates defend insights to langgraph
* adds output chunking / refinement
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
The esql types are around many packages and plugins creating problems
either with the bundle size or cyclic dependencies.
This PR is moving some of them on the esql-types package. I am going to
move more there in follow up PRs.
(I decided to rename it as it makes more sense to have a generic one for
esql rather one only for the variables)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Marco Vettorello <vettorello.marco@gmail.com>
As a first step towards making parts of streams owned by different teams
working on it, this PR gives some of the things to the logs UX team.
These routes and views are clearly associated with the workstreams the
logs UX team moves forward. It's still expected that a lot of changes
(but probably not all) will have at least a portion outside of this code
area, but it's meant to put a starting point into place to make the
separation stronger over time.
For the API it's just about certain route handlers. For the UI, the
relevant components are pulled into a separate directory to encapsulate
them somewhat.
---------
Co-authored-by: Milton Hultgren <miltonhultgren@gmail.com>
**Fixes: https://github.com/elastic/kibana/issues/206527**
**Partially addresses: https://github.com/elastic/kibana/issues/209518**
## Summary
Adds a normalization to the `filters` field in the rule diffing
calculation that omits all filter fields other than the `query` field
and the `negate` and `disabled` fields within the `meta` object. This
makes our diffing logic much more robust and resilient as we only
compare data in the rule fields that have an impact on the query itself
and not the fields that relate to UI implementation (`alias`, `key`,
etc).
### To test
- Open a prebuilt rule with `filters` in the non-customized rule
parameters (e.g. `PowerShell Script with Discovery Capabilities`)
- Edit the rule and save without editing
- The rule should remain unmodified even though more fields have been
added to the rule's `filters` field
Unless the user adds or deletes a filter on the rule, the rule should
only be marked as customized under 3 circumstances:
- The user negates the filter (adds NOT to the beginning of the filter)
- The user disables the filter
- The user changes the filter query
All other scenarios (such as adding a custom name for the filter) should
not change the rule's customized status
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
* Remove some old paths pointing to `packages/kbn-pm` (no longer
exists).
* ~Fix group and visibility for `@kbn/streams-app-wrapper-plugin`~.
(done in https://github.com/elastic/kibana/pull/212210)
* Update `scripts/relocate` logic with latest enhancements.
* Convert `@kbn/observability-synthetics-test-data` folder name to
camel-case (messes up with pre-commit hook).
## Summary
The `/packages` folder at the root of the Kibana repository used to
contain a lot of packages.
In the context of SKA, they have been gradually moved to various
locations:
* `src/platform/packages`
* `x-pack/platform/packages`
* `src/core/packages`
Currently, only `devOnly: true` packages are left in this folder. This
comprises libraries for CLI scripts as well as testing utilities.
With this PR, we are moving ~half of these packages under
`src/platform/packages/(private|shared)/`.
In particular, we are moving those packages that are being used from
platform and/or solutions.
Since they are `"devOnly": true`, this means they are ONLY used from
tests, cypress tests, storybook configs, ./scripts/ folders inside some
modules, or other non-prod-time logic. Nonetheless, they are effectively
referenced from platform and/or solutions code, hence I decided they
should be placed under `platform` folders.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This PR moves the `streams` and `streams_app` plugins into platform so
they can be used in other solutions in the future. This PR is not
actually making it available in other solutions yet since we are still
discussing the release plans.
## Inlined helpers
As discussed before, this PR inlines a couple simple helper methods for
query building, time zone normalization, a header portal helper and a
data plugin timefilter state react integration hook as there is no good
place for these outside of the observability solution.
## streams_app plugin
The streams_app plugin is not actually registering anything, instead it
simply exports a component that renders the app which needs to be
consumed by another plugin to turn it into a registered app - for now,
`observability_streams_wrapper` takes over this job.
## observability_streams_wrapper plugin
While 99% of the streams logic is moved into the
`platform/shared/streams_app`, two bits are left behind in
`observability_streams_wrapper`:
* The actual app registration
* Integration with the observability_shared `PageTemplate` component
Once we decide streams should be displayed outside of the observability
solution, it's probably not necessary anymore to decouple app definition
and registration like this because it will always be visible no matter
the solution. Once this is the case, the navigation registration can be
moved into the central `observability` plugin, like it's handled with
other apps like infra.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
These where the only packages under `src/dev/packages`.
The goal long term is to refactor the code under `src/dev` and to
migrate it to one (or multiple) packages.
Thus, we're moving them to `src/core/test-helpers/`, which already
exists and contains a bunch of test-helpers.
#### 2 packages(s) are going to be relocated:
| Id | Target folder |
| -- | ------------- |
| `@kbn/core-test-helpers-kbn-server` |
`src/core/test-helpers/kbn-server` |
| `@kbn/core-test-helpers-model-versions` |
`src/core/test-helpers/model-versions` |
<details >
<summary>Updated references</summary>
```
./.github/codeql/codeql-config.yml
./package.json
./packages/kbn-ts-projects/config-paths.json
./src/core/system/tsconfig.type_check.json
./src/core/test-helpers/kbn-server/jest.config.js
./src/core/test-helpers/model-versions/jest.config.js
./src/dev/tsconfig.type_check.json
./src/platform/packages/private/kbn-repo-packages/package-map.json
./src/plugins/kibana_usage_collection/tsconfig.type_check.json
./src/plugins/usage_collection/tsconfig.type_check.json
./tsconfig.base.json
./tsconfig.base.type_check.json
./tsconfig.refs.json
./x-pack/platform/plugins/shared/fleet/tsconfig.type_check.json
./x-pack/platform/plugins/shared/spaces/tsconfig.type_check.json
./x-pack/plugins/actions/tsconfig.type_check.json
./x-pack/plugins/alerting/tsconfig.type_check.json
./x-pack/plugins/task_manager/tsconfig.type_check.json
./yarn.lock
.github/CODEOWNERS
```
</details><details >
<summary>Updated relative paths</summary>
```
src/core/test-helpers/kbn-server/jest.config.js:12
src/core/test-helpers/kbn-server/tsconfig.json:2
src/core/test-helpers/model-versions/jest.config.js:12
src/core/test-helpers/model-versions/tsconfig.json:2
```
</details>
The storage adapter helper is a very generic package. This PR moves it
out of the observability server utils into a dedicated package to better
reflect this and to be able to use it from non-observability contexts.
The same applies to the observability es client. This PR moves it as
well and renames it to `TracedEsClient` in the same way.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR aims at relocating some of the Kibana modules (plugins and
packages) into a new folder structure, according to the _Sustainable
Kibana Architecture_ initiative.
> [!IMPORTANT]
> * We kindly ask you to:
> * Manually fix the errors in the error section below (if there are
any).
> * Search for the `packages[\/\\]` and `plugins[\/\\]` patterns in the
source code (Babel and Eslint config files), and update them
appropriately.
> * Manually review
`.buildkite/scripts/pipelines/pull_request/pipeline.ts` to ensure that
any CI pipeline customizations continue to be correctly applied after
the changed path names
> * Review all of the updated files, specially the `.ts` and `.js` files
listed in the sections below, as some of them contain relative paths
that have been updated.
> * Think of potential impact of the move, including tooling and
configuration files that can be pointing to the relocated modules. E.g.:
> * customised eslint rules
> * docs pointing to source code
> [!NOTE]
> * This PR has been auto-generated.
> * Any manual contributions will be lost if the 'relocate' script is
re-run.
> * Try to obtain the missing reviews / approvals before applying manual
fixes, and/or keep your changes in a .patch / git stash.
> * Please use
[#sustainable_kibana_architecture](https://elastic.slack.com/archives/C07TCKTA22E)
Slack channel for feedback.
Are you trying to rebase this PR to solve merge conflicts? Please follow
the steps describe
[here](https://elastic.slack.com/archives/C07TCKTA22E/p1734019532879269?thread_ts=1734019339.935419&cid=C07TCKTA22E).
#### 3 packages(s) are going to be relocated:
| Id | Target folder |
| -- | ------------- |
| `@kbn/securitysolution-data-table` |
`x-pack/solutions/security/packages/data-table` |
| `@kbn/ecs-data-quality-dashboard` |
`x-pack/solutions/security/packages/ecs-data-quality-dashboard` |
| `@kbn/security-solution-side-nav` |
`x-pack/solutions/security/packages/side-nav` |
<details >
<summary>Updated references</summary>
```
./.i18nrc.json
./package.json
./packages/kbn-ts-projects/config-paths.json
./src/platform/packages/private/kbn-repo-packages/package-map.json
./tsconfig.base.json
./tsconfig.base.type_check.json
./tsconfig.refs.json
./x-pack/solutions/security/packages/data-table/jest.config.js
./x-pack/solutions/security/packages/ecs-data-quality-dashboard/jest.config.js
./x-pack/solutions/security/packages/side-nav/jest.config.js
./yarn.lock
.github/CODEOWNERS
```
</details><details >
<summary>Updated relative paths</summary>
```
x-pack/solutions/security/packages/data-table/jest.config.js:11
x-pack/solutions/security/packages/data-table/tsconfig.json:2
x-pack/solutions/security/packages/ecs-data-quality-dashboard/jest.config.js:24
x-pack/solutions/security/packages/ecs-data-quality-dashboard/tsconfig.json:10
x-pack/solutions/security/packages/ecs-data-quality-dashboard/tsconfig.json:2
x-pack/solutions/security/packages/side-nav/jest.config.js:10
x-pack/solutions/security/packages/side-nav/tsconfig.json:2
```
</details>
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
As part of the SKA, some groups of modules have been broken-down and
relocated in different folders, negatively impacting DX. This is the
case for `serverless` modules, that are spread across:
```
- packages/serverless # storybook config
- src/platform/packages/private/serverless
- src/platform/packages/shared/serverless
```
The goal of this PR is to unify them back under
`src/platform/packages/shared/severless`.
In this scenario, I believe the simplification is worth the trade off.
CC @elastic/kibana-management @elastic/appex-sharedux WDYT?
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
As part of the SKA, some groups of modules have been broken-down and
relocated in different folders, negatively impacting DX. This is the
case for `kbn-management` modules, that are spread across:
```
- packages/kbn-management (storybook config)
- src/platform/packages/private/kbn-management
- src/platform/packages/shared/kbn-management
```
The goal of this PR is to unify them back under
`src/platform/packages/shared/kbn-management`.
In this scenario, I believe the simplification is worth the trade off.
CC @elastic/kibana-management WDYT?
Co-authored-by: Matthew Kime <matt@mattki.me>
## Summary
This PR aims at relocating some of the Kibana modules (plugins and
packages) into a new folder structure, according to the _Sustainable
Kibana Architecture_ initiative.
> [!IMPORTANT]
> * We kindly ask you to:
> * Manually fix the errors in the error section below (if there are
any).
> * Search for the `packages[\/\\]` and `plugins[\/\\]` patterns in the
source code (Babel and Eslint config files), and update them
appropriately.
> * Manually review
`.buildkite/scripts/pipelines/pull_request/pipeline.ts` to ensure that
any CI pipeline customizations continue to be correctly applied after
the changed path names
> * Review all of the updated files, specially the `.ts` and `.js` files
listed in the sections below, as some of them contain relative paths
that have been updated.
> * Think of potential impact of the move, including tooling and
configuration files that can be pointing to the relocated modules. E.g.:
> * customised eslint rules
> * docs pointing to source code
> [!NOTE]
> * This PR has been auto-generated.
> * Any manual contributions will be lost if the 'relocate' script is
re-run.
> * Try to obtain the missing reviews / approvals before applying manual
fixes, and/or keep your changes in a .patch / git stash.
> * Please use
[#sustainable_kibana_architecture](https://elastic.slack.com/archives/C07TCKTA22E)
Slack channel for feedback.
Are you trying to rebase this PR to solve merge conflicts? Please follow
the steps describe
[here](https://elastic.slack.com/archives/C07TCKTA22E/p1734019532879269?thread_ts=1734019339.935419&cid=C07TCKTA22E).
#### 4 packages(s) are going to be relocated:
| Id | Target folder |
| -- | ------------- |
| `@kbn/alerting-rule-utils` |
`x-pack/platform/packages/shared/alerting-rule-utils` |
| `@kbn/observability-alert-details` |
`x-pack/solutions/observability/packages/alert-details` |
| `@kbn/observability-alerting-test-data` |
`x-pack/solutions/observability/packages/alerting-test-data` |
| `@kbn/observability-get-padded-alert-time-range-util` |
`x-pack/solutions/observability/packages/get-padded-alert-time-range-util`
|
<details >
<summary>Updated references</summary>
```
./.i18nrc.json
./package.json
./packages/kbn-ts-projects/config-paths.json
./src/platform/packages/private/kbn-repo-packages/package-map.json
./tsconfig.base.json
./x-pack/platform/packages/shared/alerting-rule-utils/jest.config.js
./x-pack/solutions/observability/packages/alert-details/jest.config.js
./x-pack/solutions/observability/packages/alerting-test-data/jest.config.js
./x-pack/solutions/observability/packages/get-padded-alert-time-range-util/jest.config.js
./yarn.lock
.github/CODEOWNERS
```
</details><details >
<summary>Updated relative paths</summary>
```
x-pack/platform/packages/shared/alerting-rule-utils/jest.config.js:10
x-pack/platform/packages/shared/alerting-rule-utils/tsconfig.json:2
x-pack/solutions/observability/packages/alert-details/jest.config.js:10
x-pack/solutions/observability/packages/alert-details/tsconfig.json:2
x-pack/solutions/observability/packages/alerting-test-data/jest.config.js:10
x-pack/solutions/observability/packages/alerting-test-data/tsconfig.json:2
x-pack/solutions/observability/packages/get-padded-alert-time-range-util/jest.config.js:10
x-pack/solutions/observability/packages/get-padded-alert-time-range-util/tsconfig.json:2
```
</details>
## Summary
This PR aims at relocating some of the Kibana modules (plugins and
packages) into a new folder structure, according to the _Sustainable
Kibana Architecture_ initiative.
> [!IMPORTANT]
> * We kindly ask you to:
> * Manually fix the errors in the error section below (if there are
any).
> * Search for the `packages[\/\\]` and `plugins[\/\\]` patterns in the
source code (Babel and Eslint config files), and update them
appropriately.
> * Manually review
`.buildkite/scripts/pipelines/pull_request/pipeline.ts` to ensure that
any CI pipeline customizations continue to be correctly applied after
the changed path names
> * Review all of the updated files, specially the `.ts` and `.js` files
listed in the sections below, as some of them contain relative paths
that have been updated.
> * Think of potential impact of the move, including tooling and
configuration files that can be pointing to the relocated modules. E.g.:
> * customised eslint rules
> * docs pointing to source code
> [!NOTE]
> * This PR has been auto-generated.
> * Any manual contributions will be lost if the 'relocate' script is
re-run.
> * Try to obtain the missing reviews / approvals before applying manual
fixes, and/or keep your changes in a .patch / git stash.
> * Please use
[#sustainable_kibana_architecture](https://elastic.slack.com/archives/C07TCKTA22E)
Slack channel for feedback.
Are you trying to rebase this PR to solve merge conflicts? Please follow
the steps describe
[here](https://elastic.slack.com/archives/C07TCKTA22E/p1734019532879269?thread_ts=1734019339.935419&cid=C07TCKTA22E).
#### 3 packages(s) are going to be relocated:
| Id | Target folder |
| -- | ------------- |
| `@kbn/response-ops-alerts-apis` |
`src/platform/packages/shared/response-ops/alerts-apis` |
| `@kbn/response-ops-alerts-fields-browser` |
`src/platform/packages/shared/response-ops/alerts-fields-browser` |
| `@kbn/response-ops-alerts-table` |
`src/platform/packages/shared/response-ops/alerts-table` |
<details >
<summary>Updated references</summary>
```
./package.json
./packages/kbn-ts-projects/config-paths.json
./src/platform/packages/private/kbn-repo-packages/package-map.json
./src/platform/packages/shared/response-ops/alerts-apis/jest.config.js
./src/platform/packages/shared/response-ops/alerts-fields-browser/jest.config.js
./src/platform/packages/shared/response-ops/alerts-table/jest.config.js
./tsconfig.base.json
./yarn.lock
.github/CODEOWNERS
```
</details><details >
<summary>Updated relative paths</summary>
```
src/platform/packages/shared/response-ops/alerts-apis/jest.config.js:12
src/platform/packages/shared/response-ops/alerts-apis/tsconfig.json:2
src/platform/packages/shared/response-ops/alerts-fields-browser/jest.config.js:12
src/platform/packages/shared/response-ops/alerts-fields-browser/tsconfig.json:2
src/platform/packages/shared/response-ops/alerts-table/jest.config.js:12
src/platform/packages/shared/response-ops/alerts-table/tsconfig.json:2
```
</details>
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Umberto Pepato <umbopepato@users.noreply.github.com>
## Summary
This PR aims at relocating some of the Kibana modules (plugins and
packages) into a new folder structure, according to the _Sustainable
Kibana Architecture_ initiative.
> [!IMPORTANT]
> * We kindly ask you to:
> * Manually fix the errors in the error section below (if there are
any).
> * Search for the `packages[\/\\]` and `plugins[\/\\]` patterns in the
source code (Babel and Eslint config files), and update them
appropriately.
> * Manually review
`.buildkite/scripts/pipelines/pull_request/pipeline.ts` to ensure that
any CI pipeline customizations continue to be correctly applied after
the changed path names
> * Review all of the updated files, specially the `.ts` and `.js` files
listed in the sections below, as some of them contain relative paths
that have been updated.
> * Think of potential impact of the move, including tooling and
configuration files that can be pointing to the relocated modules. E.g.:
> * customised eslint rules
> * docs pointing to source code
> [!NOTE]
> * This PR has been auto-generated.
> * Any manual contributions will be lost if the 'relocate' script is
re-run.
> * Try to obtain the missing reviews / approvals before applying manual
fixes, and/or keep your changes in a .patch / git stash.
> * Please use
[#sustainable_kibana_architecture](https://elastic.slack.com/archives/C07TCKTA22E)
Slack channel for feedback.
Are you trying to rebase this PR to solve merge conflicts? Please follow
the steps describe
[here](https://elastic.slack.com/archives/C07TCKTA22E/p1734019532879269?thread_ts=1734019339.935419&cid=C07TCKTA22E).
#### 1 packages(s) are going to be relocated:
| Id | Target folder |
| -- | ------------- |
| `@kbn/logs-overview` | `x-pack/platform/packages/shared/logs-overview`
|
<details >
<summary>Updated references</summary>
```
./package.json
./packages/kbn-ts-projects/config-paths.json
./src/platform/packages/private/kbn-repo-packages/package-map.json
./tsconfig.base.json
./x-pack/.i18nrc.json
./x-pack/platform/packages/shared/logs-overview/jest.config.js
./yarn.lock
.github/CODEOWNERS
```
</details><details >
<summary>Updated relative paths</summary>
```
x-pack/platform/packages/shared/logs-overview/jest.config.js:10
x-pack/platform/packages/shared/logs-overview/tsconfig.json:2
```
</details>
## Summary
This PR aims at relocating some of the Kibana modules (plugins and
packages) into a new folder structure, according to the _Sustainable
Kibana Architecture_ initiative.
> [!IMPORTANT]
> * We kindly ask you to:
> * Manually fix the errors in the error section below (if there are
any).
> * Search for the `packages[\/\\]` and `plugins[\/\\]` patterns in the
source code (Babel and Eslint config files), and update them
appropriately.
> * Manually review
`.buildkite/scripts/pipelines/pull_request/pipeline.ts` to ensure that
any CI pipeline customizations continue to be correctly applied after
the changed path names
> * Review all of the updated files, specially the `.ts` and `.js` files
listed in the sections below, as some of them contain relative paths
that have been updated.
> * Think of potential impact of the move, including tooling and
configuration files that can be pointing to the relocated modules. E.g.:
> * customised eslint rules
> * docs pointing to source code
> [!NOTE]
> * This PR has been auto-generated.
> * Any manual contributions will be lost if the 'relocate' script is
re-run.
> * Try to obtain the missing reviews / approvals before applying manual
fixes, and/or keep your changes in a .patch / git stash.
> * Please use
[#sustainable_kibana_architecture](https://elastic.slack.com/archives/C07TCKTA22E)
Slack channel for feedback.
Are you trying to rebase this PR to solve merge conflicts? Please follow
the steps describe
[here](https://elastic.slack.com/archives/C07TCKTA22E/p1734019532879269?thread_ts=1734019339.935419&cid=C07TCKTA22E).
#### 1 packages(s) are going to be relocated:
| Id | Target folder |
| -- | ------------- |
| `@kbn/search-shared-ui` | `x-pack/solutions/search/packages/shared-ui`
|
<details >
<summary>Updated references</summary>
```
./package.json
./packages/kbn-relocate/utils/transforms.ts
./packages/kbn-ts-projects/config-paths.json
./src/platform/packages/private/kbn-repo-packages/package-map.json
./tsconfig.base.json
./x-pack/.i18nrc.json
./x-pack/solutions/search/packages/shared-ui/jest.config.js
./yarn.lock
.github/CODEOWNERS
```
</details><details >
<summary>Updated relative paths</summary>
```
x-pack/solutions/search/packages/shared-ui/jest.config.js:14
x-pack/solutions/search/packages/shared-ui/tsconfig.json:2
```
</details>
## Summary
This PR aims at relocating some of the Kibana modules (plugins and
packages) into a new folder structure, according to the _Sustainable
Kibana Architecture_ initiative.
> [!IMPORTANT]
> * We kindly ask you to:
> * Manually fix the errors in the error section below (if there are
any).
> * Search for the `packages[\/\\]` and `plugins[\/\\]` patterns in the
source code (Babel and Eslint config files), and update them
appropriately.
> * Manually review
`.buildkite/scripts/pipelines/pull_request/pipeline.ts` to ensure that
any CI pipeline customizations continue to be correctly applied after
the changed path names
> * Review all of the updated files, specially the `.ts` and `.js` files
listed in the sections below, as some of them contain relative paths
that have been updated.
> * Think of potential impact of the move, including tooling and
configuration files that can be pointing to the relocated modules. E.g.:
> * customised eslint rules
> * docs pointing to source code
> [!NOTE]
> * This PR has been auto-generated.
> * Any manual contributions will be lost if the 'relocate' script is
re-run.
> * Try to obtain the missing reviews / approvals before applying manual
fixes, and/or keep your changes in a .patch / git stash.
> * Please use
[#sustainable_kibana_architecture](https://elastic.slack.com/archives/C07TCKTA22E)
Slack channel for feedback.
Are you trying to rebase this PR to solve merge conflicts? Please follow
the steps describe
[here](https://elastic.slack.com/archives/C07TCKTA22E/p1734019532879269?thread_ts=1734019339.935419&cid=C07TCKTA22E).
#### 2 packages(s) are going to be relocated:
| Id | Target folder |
| -- | ------------- |
| `@kbn/observability-ai-common` |
`x-pack/solutions/observability/packages/observability-ai/observability-ai-common`
|
| `@kbn/observability-ai-server` |
`x-pack/solutions/observability/packages/observability-ai/observability-ai-server`
|
<details >
<summary>Updated references</summary>
```
./package.json
./packages/kbn-ts-projects/config-paths.json
./src/platform/packages/private/kbn-repo-packages/package-map.json
./tsconfig.base.json
./x-pack/solutions/observability/packages/observability-ai/observability-ai-common/jest.config.js
./x-pack/solutions/observability/packages/observability-ai/observability-ai-server/jest.config.js
./yarn.lock
.github/CODEOWNERS
```
</details><details >
<summary>Updated relative paths</summary>
```
x-pack/solutions/observability/packages/observability-ai/observability-ai-common/jest.config.js:10
x-pack/solutions/observability/packages/observability-ai/observability-ai-common/tsconfig.json:2
x-pack/solutions/observability/packages/observability-ai/observability-ai-server/jest.config.js:10
x-pack/solutions/observability/packages/observability-ai/observability-ai-server/tsconfig.json:2
```
</details>
## Summary
This PR aims at relocating some of the Kibana modules (plugins and
packages) into a new folder structure, according to the _Sustainable
Kibana Architecture_ initiative.
> [!IMPORTANT]
> * We kindly ask you to:
> * Manually fix the errors in the error section below (if there are
any).
> * Search for the `packages[\/\\]` and `plugins[\/\\]` patterns in the
source code (Babel and Eslint config files), and update them
appropriately.
> * Manually review
`.buildkite/scripts/pipelines/pull_request/pipeline.ts` to ensure that
any CI pipeline customizations continue to be correctly applied after
the changed path names
> * Review all of the updated files, specially the `.ts` and `.js` files
listed in the sections below, as some of them contain relative paths
that have been updated.
> * Think of potential impact of the move, including tooling and
configuration files that can be pointing to the relocated modules. E.g.:
> * customised eslint rules
> * docs pointing to source code
> [!NOTE]
> * This PR has been auto-generated.
> * Any manual contributions will be lost if the 'relocate' script is
re-run.
> * Try to obtain the missing reviews / approvals before applying manual
fixes, and/or keep your changes in a .patch / git stash.
> * Please use
[#sustainable_kibana_architecture](https://elastic.slack.com/archives/C07TCKTA22E)
Slack channel for feedback.
Are you trying to rebase this PR to solve merge conflicts? Please follow
the steps describe
[here](https://elastic.slack.com/archives/C07TCKTA22E/p1734019532879269?thread_ts=1734019339.935419&cid=C07TCKTA22E).
#### 3 packages(s) are going to be relocated:
| Id | Target folder |
| -- | ------------- |
| `@kbn/observability-utils-browser` |
`x-pack/solutions/observability/packages/utils-browser` |
| `@kbn/observability-utils-common` |
`x-pack/solutions/observability/packages/utils-common` |
| `@kbn/observability-utils-server` |
`x-pack/solutions/observability/packages/utils-server` |
<details >
<summary>Updated references</summary>
```
./package.json
./packages/kbn-ts-projects/config-paths.json
./src/platform/packages/private/kbn-repo-packages/package-map.json
./tsconfig.base.json
./x-pack/solutions/observability/packages/utils-browser/jest.config.js
./x-pack/solutions/observability/packages/utils-common/jest.config.js
./x-pack/solutions/observability/packages/utils-server/jest.config.js
./x-pack/solutions/observability/packages/utils-server/jest.integration.config.js
./yarn.lock
.github/CODEOWNERS
```
</details><details >
<summary>Updated relative paths</summary>
```
x-pack/solutions/observability/packages/utils-browser/jest.config.js:10
x-pack/solutions/observability/packages/utils-browser/tsconfig.json:2
x-pack/solutions/observability/packages/utils-common/jest.config.js:10
x-pack/solutions/observability/packages/utils-common/tsconfig.json:2
x-pack/solutions/observability/packages/utils-server/jest.config.js:10
x-pack/solutions/observability/packages/utils-server/jest.integration.config.js:10
x-pack/solutions/observability/packages/utils-server/tsconfig.json:2
```
</details>
Closes https://github.com/elastic/kibana/issues/208016
Adds a new dataset for the tag-association test (didn't find a good
existing one)
For the MKI flakiness, I don't think this is related to this test but
rather another test leaking SOs. I added some robustness against this,
but ideally we fix this in the offending test. Maybe we should add a
little something that checks for leaky state after a suite closes and
fails if there is something? That goes beyond the scope of this issue
though.
### Summary
- Closes#89741
This PR contains the resulting work of a massive effort that ports our
on top bundler abstraction (called @kbn/optimizer) from Webpack v4 into
Webpack v5. It's essential in terms of long term maintenance since v4
was not receiving updates any longer but will also unblock some new
features that could be beneficial for our future DevEx endeavours.
Next you can find a small list of all the accomplished tasks on this
journey.
### Completed Tasks
- [x] Upgrade dependencies to match the ones on webpack v5
- [x] Fix null-loader usages
- [x] Fix raw-loader usages
- [x] Fix file-loader usages
- [x] Fix url-loader usages
- [x] Fix `@kbn/optimizer-webpack-helpers` to support webpack v5
- [x] Adopt previous webpack v4 polyfill-all strategy with
node-polyfill-webpack-plugin
- [x] Fix theme-loader on @kbn/optimizer
- [x] Migrate configurations and ad-hoc loader options on all webpack
configs from v4 to v5
- [x] Fix @kbn/test jest resolver for file-loader cases
- [x] Migrate public-path loader on UiSharedDeps
- [x] Fix all usages of webpack-merge
- [x] Migrate BundleRemoteModule
- [x] Migrate BundleRemotesPlugin
- [x] Correctly migrate PopulateBundleCachePlugin
- [x] Correctly migrate BundleMetricsPlugin
- [x] Check if the profiling plugins still work (--profile flag)
- [x] Recover if possible the previous webpack v4 cacheGroup chunks
rename to something like `data.plugin.chunk.0.js`
- [x] Run `/ci` and make sure we get our first green CI, otherwise work
on the errors until we do
- [x] Profile and solve bottlenecks until we get a cold build
performance similar to the one we had on webpack v4 (`node
scripts/build_kibana_platform_plugins --no-cache`).
- [x] OpenSSL Legacy Warnings: try to remove `--openssl-legacy-provider
` flags
- [x] Add Webpack to Renovate config
- [x] Explore removing `NodePolyfillPlugin`
([here](https://www.npmjs.com/package/node-polyfill-webpack-plugin)) and
add each polyfill needed individually per each webpack config to check
if we get smaller bundles. If we do it's better to go with the case by
case need approach instead of deploying a bunch of polyfills with
NodePolyfillPlugin. As another alternative, create a custom smaller
plugin with only the union of all needed polyfills.
- [x] Evaluate if we want to touch the resolutions on mainFields and
conditionNames
- [x] Understand why `@import 'src/core/public/mixins'` does not work
anymore (not a problem, we should use relative paths anyway but we want
to track why it changed from v4 to v5)
- [x] BUG: Child compilers are having errors hidden and/or changed from
error to warning
- [x] Fix license check for
[Artistic-2.0](https://spdx.org/licenses/Artistic-2.0.html) is the
license for
[domain-browser](https://github.com/bevry/domain-browser?tab=License-1-ov-file).
This package is a dependency of
[NodePolyfillPlugin](https://www.npmjs.com/package/node-polyfill-webpack-plugin).
Artistic 2.0 license is [classified as
yellow](https://github.com/elastic/open-source/blob/main/elastic-product-policy.md#yellow-list)
and should only be used for dev dependencies.
- [x] Make sure `resourceQuery: { not: /raw/ }` is not necessary on
other webpack configs like storybook one
- [x] Find what is being wrongly removed by usedExports optimization;
hint: I believe it is identifying a lot of exports inside the sync entry
of plugins as unused exports and removing them. Then `__kbnBootstrap__`
can't be found
- [x] Rebalance @kbn/optimizer pickMaxWorkerCount
- [x] Re-open the issue to fix sass-warnings
[#190345](https://github.com/elastic/kibana/issues/190345) or downgrade
sass-loader to v10
- [x] Remove previous esm no parse rules
- [x] Confirm esm support is working
- [x] Confirm console override is needed
- [x] Confirm react prod builds on ui shared deps for distributable
- [x] Remove customization for
[xyflow](https://github.com/xyflow/xyflow) from webpack configs
- [x] Clean all the code
- [x] Make sure collected metrics from stats are still aligned with what
we were collecting before; also verify if the modules used for optimizer
caches etc are well generated (@kbn/node-libs-browser)
- [x] Fix watch performance
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Brad White <brad.white@elastic.co>
## Summary
Both files were incorrectly assigned in
https://github.com/elastic/kibana/pull/207969. This PR corrects the
assignment to the security solution team.
- /x-pack/test_serverless/functional/test_suites/security/constants.ts
- /x-pack/test_serverless/api_integration/test_suites/security
Also, the ownership assinment of
`/x-pack/test_serverless/**/test_suites/security/platform_security` was
moved below
`/x-pack/test_serverless/api_integration/test_suites/security`, to
maintain hierarchical ownership.
## Summary
This PR turns the AlertsTable into a standalone component, making it
independent from the `TriggersActionsUI` plugin.
#### Removes the alerts table registry
All configuration is now managed through the AlertsTable component
props. Shared configurations are handled by giving consumers the ability
to directly provide alerts table wrapper components (see for example the
`renderAlertsTable` prop of `getCases`).
#### Moves the alerts table to dedicated package(s)
Following the feature-driven structure we're introducing for ResponseOps
(alerting) client-side packages:
- `@kbn/response-ops-alerts-table`
- `@kbn/response-ops-alerts-apis`
- `@kbn/response-ops-alerts-fields-browser`
#### Initial work on improving composition and organization
- Reorganizes the table code into a by-entity-type folder structure
(`components/`, `hooks/`, ...)
- Simplifies some components and breaks into smaller units when possible
## To verify
For consumers of the alerts table:
- Check that all your tables have the same behavior as before (columns,
sort, row actions, bulk actions, etc.)
- Check that your "shared" tables (i.e. cases alerts view in O11y and
Security) have the expected configuration and behavior
> [!WARNING]
> This PR moves a lot of files. Git might not always recognize the
correct delete/add file pairs. If you see weird diffs feel free to reach
out for help!
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
### Identify risks
| Risk | Description | Severity | Mitigation |
|---|---|---|---|
| Table misconfigurations | Some table configurations might slightly
differ from the previous AlertsTableRegistry-backed version | Low |
Quick fix |
## References
Closes#195180
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Christos Nasikas <xristosnasikas@gmail.com>
## Summary
`@kbn/scout-oblt` is a test library that extends `@kbn/scout` with test
helpers specifically designed to test `Observability` applications in
Kibana. All Oblt plugins should only import from `@kbn/scout-oblt`
Its primary goal is to simplify the test development experience for
teams working on `Observability` plugins by providing custom Playwright
fixtures, page objects, and utilities tailored for Observability-related
testing scenarios.
Contributing:
- when Fixture/Page Object is sharable across all Solutions and Platform
(`fleetApi` fixture), it should be added in `@kbn/scout`
- when Fixture/Page Object is Oblt-specific but is shared across tests
under the multiple plugins (`OnboardingHome` page), it should be added
in `@kbn/scout-oblt`
- when Fixture/Page Object is only used in a single plugin (`onboarding`
internal APIs ?), it should be added in this plugin.
I also re-worked existing tests with few ideas in mind:
- Scout is **e2e testing tool** and should target primary e2e test
scenarios; We have _API integration tests_ to test multiple short
scenarios for APIs behavior (response, status code) and _jest/React
testing library_ to test components in isolation (elements rendering,
fields validation). Doing all the testing with e2e tool like Playwright
will dramatically affect cost efficiency and stability of tests, but
also slows overall CI execution and PRs delivery. The goal is to follow
testing pyramid and keep in mind its principles.
- We on purpose spin up new browser context for each `test` block to
make sure our **tests are independent**. Having too many short `test`
blocks in the file significantly slows down the execution: every block
triggers browser context, saml authentication, adding/removing Fleet
integrations (each call up to 2 seconds) and other beforeEach/afterEach
hooks. Real browser-based testing is expensive. It is not about putting
every step into 1 `test` block, but also not a Jest unit-test-style
design. When it is possible to group similar actions on the same page
and if it is a part of the same user flow - we should do it. It also
doesn't bring the testing value repeating the same UI steps multiple
times in different scenarios. _Our CI costs are critical to cut when it
is possible_
- Avoid **nesting describe** blocks: it complicates test readability and
also complicates for CI bot to properly skip the failing block (it will
skip the top level one). We encourage **Scout parallel test execution**
based on running test spec files in multiple workers, not the `test`
blocks within the same file. Having too many `test` blocks in the same
file will be slowly run in the single thread and in case of flakiness,
it means Team lose more test coverage than they probably expect.
Before (**59** test blocks - **8-8.5 min** per distro):
<img width="1709" alt="Screenshot 2025-02-08 at 18 01 40"
src="https://github.com/user-attachments/assets/5fd65a1c-85f9-4594-9dae-3f8e99a005ab"
/>
After (**15** test blocks - **3.5-4 min** per distro):
<img width="1578" alt="Screenshot 2025-02-10 at 18 14 42"
src="https://github.com/user-attachments/assets/6846898f-7dd2-4f6b-8bc5-d06741b0b120"
/>
For reviewers: updated tests are possible to run in 2 parallel workers
against the same Kibana/ES instance and run time is dropping to **2.5-3
min** 🚀 . It is up to UX-Logs team to decide if you want to keep
parallel run (new tests can be added either to parallel or sequential
run)
<img width="1578" alt="Screenshot 2025-02-11 at 12 14 30"
src="https://github.com/user-attachments/assets/e94113f2-d7f1-470e-a6d5-cb5154d99c41"
/>
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Resolves#209261
## Summary
Removes the code used to render Logs Explorer. This does not result in
any functional changes.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
* Categorise and move `@kbn/timelines-plugin` as _platform/shared_,
target location: `x-pack/platform/plugins/shared/timelines`.
This helps reduce the scope of the illegal dependencies from `osquery`
plugin towards _security/private_ code.
cc @tomsonpl
* Simplify path and rename `@kbn/observability-alerting-rule-utils`
(platform/shared):
```
# Before
@kbn/observability-alerting-rule-utils
x-pack/platform/packages/shared/observability/alerting_rule_utils/
# After
@kbn/alerting-rule-utils
x-pack/platform/packages/shared/alerting_rule_utils/
```
* Simplify path and rename `@kbn/observability-logs-overview`
(platform/shared):
```
# Before
@kbn/observability-logs-overview
x-pack/platform/packages/shared/observability/logs_overview/
# After
@kbn/logs-overview
x-pack/platform/packages/shared/logs_overview/
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
As titled. The Project Switcher was created in #155582 as a means of
allowing a developer to switch between project types without having to
restart Kibana. Since that first PR, Kibana *must* be started with a
particular project configuration in mind-- the switcher now does
nothing. This PR removes it and deprecates the config keys.
## Summary
The other day when the https://github.com/elastic/kibana/pull/208546 was
opened, no required review was triggered for the changes in
`x-pack/test/security_solution_cypress/cypress/tasks/api_calls/common.ts`.
This is a problem, since any change in that file can have a huge impact
in the overall Cypress test suite.
In this PR we are adding security engineering productivity as owners of
that file to make sure that changes there are properly reviewed.
## Summary
Part of https://github.com/elastic/kibana/issues/206710
This PR introduces the `InferenceChatModel` class, which is a langchain
chatModel utilizing the inference APIs (`chatComplete`) under the hood.
Creating instances of `InferenceChatModel` can either be done by
manually importing the class from the new `@kbn/inference-langchain`
package, or by using the new `createChatModel` API exposes from the
inference plugin's start contract.
The main upside of using this chatModel is that the unification and
normalization layers are already being taken care of by the inference
plugin, making sure that the underlying models are being used with the
exact same capabilities. More details on the upsides and reasoning in
the associated issue.
### Usage
Usage is very straightforward
```ts
const chatModel = await inferenceStart.getChatModel({
request,
connectorId: myInferenceConnectorId,
chatModelOptions: {
temperature: 0.2,
},
});
// just use it as another langchain chatModel, e.g.
const response = await chatModel.stream('What is Kibana?');
for await (const chunk of response) {
// do something with the chunk
}
```
### Important
This PR is only adding the implementation, and not wiring it anywhere or
using it in any existing code. This is meant to be done in a later
stage. Merging that implementation first will allow to have distinct PRs
for the integration with search (playground) and security (assistant +
other workflows), with proper testing
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Often we've failures on the MKI FTR runs in Buildkite, that are due to
our roles files being out of sync.
This is an attempt to get ahead of the problem, or to at least be
notified that a change is coming.
## Summary
Add MapsPage to the scout core to be re-used by others.
Add rudimentary docs to show how to run these tests.
Added a `waitForRender` method.
Add test id.
---------
Co-authored-by: Robert Oskamp <traeluki@gmail.com>
Co-authored-by: Nick Partridge <nicholas.partridge@elastic.co>
Co-authored-by: Nick Partridge <nick.ryan.partridge@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
A minimal version of the file upload tool which can be triggered via a
uiAction.
The trigger takes a callback to enable subsequent actions after the
upload. This callback receives information about the upload, the index
and data view created and information about the files:
```
{
"index": "test9",
"dataView": {
"id": "a870ef68-a624-4df1-9d5d-fa62b75dd297",
"title": ""
},
"files": [
{
"fileName": "farequote-tiny.csv",
"docCount": 20,
"fileFormat": "delimited"
},
{
"fileName": "farequote.csv",
"docCount": 86275,
"fileFormat": "delimited"
}
]
}
```
If `autoAddInference` is set with the name of an inference endpoint
(`autoAddInference: '.elser-2-elasticsearch'`) the tool with
automatically add a `semantic_text` to the mappings for tika files (pdf,
txt docx)
Currently embedded in the search app's home page and playground. In
playgroubnd, after upload is complete and the flyout closed, the newly
created index will be selected.
https://github.com/user-attachments/assets/0589fa02-fb0e-400b-8e74-1eb9a993c6ba
Multiple files can be uploaded at once. They must be of the same file
format and the mappings cannot clash.
If more than once file has the same field but they are of different
types, the files are considered incompatible.
<img width="612" alt="image"
src="https://github.com/user-attachments/assets/67307fd0-7d10-4eab-9e72-df133ebddcfe"
/>
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
