## Summary
This PR updates `DEFAULT_THEME_TAGS` used to determine what theme tags
are bundled in Kibana by default to only include the Borealis theme,
specifically `borealislight` and `borealisdark` theme tags. This change
is expected to decrease bundle sizes significantly and get back to
bundling a single theme, not two (4 → 2 theme tags).
Now that Serverless, `9.0`, and `main` all run with Borealis, there's no
risk in removing Amsterdam from the bundle and decreasing Kibana bundle
sizes.
We need to keep the feature flag in code for the time being to easily
test future Borealis iterations.
Amsterdam will still be available as an opt-in theme and is meant to be
used locally when testing changes to be backported to 8.x versions that
use Amsterdam. To do so, Kibana needs to be started/built with
`KBN_OPTIMIZER_THEMES` environment variable set and the feature flag
overridden in `kibana.dev.yml`.
```yml
# config/kibana.dev.yml
feature_flags.overrides.coreRendering.defaultThemeName: amsterdam
```
```shell
# Run dev server with both borealis and Amsterdam theme tags
KBN_OPTIMIZER_THEMES="borealislight,borealisdark,v8light,v8dark" yarn start
```
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Remove extraneous dependencies:
* `canvas` was depending on 'webpack' purely for a type (dev-time).
* `@kbn/optimizer-webpack-helpers` (canvas depends on it 🤨) was
depending on 'webpack' solely for a function that could be defined in
`@kbn/optimizer` (devOnly).
## Summary
Closes https://github.com/elastic/ingest-dev/issues/4722
### Implementation checklist
- [x] Handle fetching agent policies and agents at scale
- [x] Only consider active agents for upgrade
- [x] Agents already on or upgrading to target version are included in
the count but not considered for upgrade
- [x] Agents stuck in updating are considered for upgrade
- [x] Bulk upgrade actions triggered by the task have an added
`isAutomatic:true` flag
- [x] Use rollout duration to spread bulk upgrade in time (1h or longer
depending on agent count)
### Testing
- This should be tested with real Elastic Agents (that will upgrade and
have `upgrade_details`).
- Edit the task interval in order to test how the task logic handles
agents already upgrading.
- Edit the agents batch size in order to test how the task logic handles
agents at scale.
- We should also check that space awareness is respected if enabled.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
Risk of incorrectly triggering agent upgrades. Probability should be
very low if the agent policy does not have `required_versions` set.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Updating the ES client to 9.0.
Resolves#116102
## What changes?
**Breaking change**: `body` has been removed.
Most of the changes are about bringing all the content inside the body
as a root attribute to the API params:
```diff
const response = await client.search({
index: 'test',
- body: {
query: {
match_all: {}
}
- }
})
```
For this reason, enabling the "Hide whitespace changes" option when
reviewing is recommended.
Some exceptions to this rule:
* Bulk APIs replace the `body` array with `operations` array (direct
replacement)
* Index Put Settings API replace `body` array with `settings` (direct
replacement)
* Msearch replaces the `body` array with `searches` array (direct
replacement)
* Document Index API replaces `body` with `document` (direct
replacement)
* Create Repository replaces `body` with `repository` (direct
replacement)
Because of a known issue in the client
(https://github.com/elastic/elasticsearch-js/issues/2584), there's still
an escape hatch to send data in the body in case the specific use case
requires it via `// @ts-expect-error elasticsearch@9.0.0
https://github.com/elastic/elasticsearch-js/issues/2584`, but it
shouldn't be abused because we lose types. In this PR we've used it in
those scenarios where we reuse the response of a GET as the body of a
PUT/POST.
### Other changes
* `estypes` can be imported from the root of the library as `import type
{ estypes } from '@elastic/elasticsearch';`
* `estypesWithBody` have been removed
* `requestTimeout`'s 30s default has been removed in the client. This PR
explicitly adds the setting in all client usages.
### Identify risks
- [x] The client places unknown properties as querystring, risking body
params leaking there, and causing 400 errors from ES => Solved by
forcing `body` usage there via `// @ts-expect-error elasticsearch@9.0.0
https://github.com/elastic/elasticsearch-js/issues/2584`. The next
version of the client will address this.
- [x] We need to run the MKI tests to make sure that we're not breaking
anything there =>
https://elastic.slack.com/archives/C04HT4P1YS3/p1739528112482629?thread_ts=1739480136.231439&cid=C04HT4P1YS3
---------
Co-authored-by: Gloria Hornero <gloria.hornero@elastic.co>
## Summary
* Remove some old paths pointing to `packages/kbn-pm` (no longer
exists).
* ~Fix group and visibility for `@kbn/streams-app-wrapper-plugin`~.
(done in https://github.com/elastic/kibana/pull/212210)
* Update `scripts/relocate` logic with latest enhancements.
* Convert `@kbn/observability-synthetics-test-data` folder name to
camel-case (messes up with pre-commit hook).
## Summary
The `/packages` folder at the root of the Kibana repository used to
contain a lot of packages.
In the context of SKA, they have been gradually moved to various
locations:
* `src/platform/packages`
* `x-pack/platform/packages`
* `src/core/packages`
Currently, only `devOnly: true` packages are left in this folder. This
comprises libraries for CLI scripts as well as testing utilities.
With this PR, we are moving ~half of these packages under
`src/platform/packages/(private|shared)/`.
In particular, we are moving those packages that are being used from
platform and/or solutions.
Since they are `"devOnly": true`, this means they are ONLY used from
tests, cypress tests, storybook configs, ./scripts/ folders inside some
modules, or other non-prod-time logic. Nonetheless, they are effectively
referenced from platform and/or solutions code, hence I decided they
should be placed under `platform` folders.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR improves how **Synthtrace** resolves the Kibana URL when only
`--target` (Elasticsearch) is provided or when neither `--target` nor
`--kibana` is specified. The CLI now attempts to **automatically
discover** the appropriate URLs based on the provided arguments.
Some adjustments were made to improve this discovery process, especially
when running **locally in Serverless mode**, where Kibana may be using
`http`, while Elasticsearch (ES) is on `https`. Additionally,
self-signed certificates do not work with the IP address `127.0.0.1`, so
this PR defaults to `localhost` and warns the user if `127.0.0.1` is
detected in Serverless mode.
### **Improvements**
- If either of `--target` or `--kibana` or neither provided, the CLI
attempts to **discovers the URLs** dynamically now in both Stateful and
Serverless.
- Defaults to `localhost` instead of `127.0.0.1` to avoid SSL
certificate issues.
- Provides a **clear error message and hint** when Kibana and ES use
different protocols (http vs https) and either or both are unreachable.
### **Expected Behavior After This PR**
These commands should now work **seamlessly** in both **local Stateful**
and **Serverless** modes:
```sh
✗ node scripts/synthtrace simple_logs
```
For **Serverless mode**, these also work:
```sh
✗ node scripts/synthtrace simple_logs --kibana=http://elastic_serverless:changeme@localhost:5601
```
```sh
✗ node scripts/synthtrace simple_logs --target=https://elastic_serverless:changeme@localhost:9200 --kibana=http://elastic_serverless:changeme@localhost:5601
```
### **(Side Note) Serverless Kibana with SSL Disabled**
However, the following command will **fail** with an error message if
Kibana is running without SSL, while Elasticsearch is using `https`:
```sh
✗ node scripts/synthtrace simple_logs --target=https://elastic_serverless:changeme@localhost:9200
```
#### **Error Output:**
```sh
Loading scenario from kibana/packages/kbn-apm-synthtrace/src/scenarios/simple_logs.ts
Error: Could not connect to Kibana. request to https://elastic_serverless:changeme@localhost:5601/ failed, reason: write EPROTO 400882F501000000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:355:
If your Kibana URL differs, consider using the '--kibana' parameter to customize it.
```
**Solution:**
If you must have to provide `--target` (non defaults), also provide
`--kibana` or start Kibana with SSL enabled.
```sh
✗ yarn start --serverless=oblt --ssl
```
## Summary
In #211918 I added config validation check to skip run if there are no
tests in playwright config.
It turned out that Playwright init reporters even when `--list` command
is passed and no tests are executed, that lead to Scout reports being
loaded and then causing reporter error when the other command runs the
tests:
```
proc [playwright] info Calling save with destination: /Users/dmle/github/kibana/.scout/reports/scout-playwright-9518363d47816953
proc [playwright] ERROR Error: Save destination path '/Users/dmle/github/kibana/.scout/reports/scout-playwright-9518363d47816953' already exists
proc [playwright] at ScoutEventsReport.save (/Users/dmle/github/kibana/packages/kbn-scout-reporting/src/reporting/report/events/report.ts:56:13)
proc [playwright] at ScoutPlaywrightReporter.onEnd (/Users/dmle/github/kibana/packages/kbn-scout-reporting/src/reporting/playwright/events/playwright_reporter.ts:277:19)
proc [playwright] at ReporterV2Wrapper.onEnd (/Users/dmle/github/kibana/node_modules/playwright/lib/reporters/reporterV2.js:91:165)
proc [playwright] at /Users/dmle/github/kibana/node_modules/playwright/lib/reporters/multiplexer.js:71:117
proc [playwright] at wrapAsync (/Users/dmle/github/kibana/node_modules/playwright/lib/reporters/multiplexer.js:112:18)
proc [playwright] at Multiplexer.onEnd (/Users/dmle/github/kibana/node_modules/playwright/lib/reporters/multiplexer.js:69:31)
proc [playwright] at InternalReporter.onEnd (/Users/dmle/github/kibana/node_modules/playwright/lib/reporters/internalReporter.js:77:12)
proc [playwright] at finishTaskRun (/Users/dmle/github/kibana/node_modules/playwright/lib/runner/tasks.js:90:26)
proc [playwright] at runTasks (/Users/dmle/github/kibana/node_modules/playwright/lib/runner/tasks.js:73:10)
proc [playwright] at Runner.runAllTests (/Users/dmle/github/kibana/node_modules/playwright/lib/runner/runner.js:72:20)
proc [playwright] at runTests (/Users/dmle/github/kibana/node_modules/playwright/lib/program.js:211:18)
proc [playwright] at t.<anonymous> (/Users/dmle/github/kibana/node_modules/playwright/lib/program.js:54:7)
```
The simplest solution is to explicitly disable Scout reporter for config
validation command.
This PR moves the `streams` and `streams_app` plugins into platform so
they can be used in other solutions in the future. This PR is not
actually making it available in other solutions yet since we are still
discussing the release plans.
## Inlined helpers
As discussed before, this PR inlines a couple simple helper methods for
query building, time zone normalization, a header portal helper and a
data plugin timefilter state react integration hook as there is no good
place for these outside of the observability solution.
## streams_app plugin
The streams_app plugin is not actually registering anything, instead it
simply exports a component that renders the app which needs to be
consumed by another plugin to turn it into a registered app - for now,
`observability_streams_wrapper` takes over this job.
## observability_streams_wrapper plugin
While 99% of the streams logic is moved into the
`platform/shared/streams_app`, two bits are left behind in
`observability_streams_wrapper`:
* The actual app registration
* Integration with the observability_shared `PageTemplate` component
Once we decide streams should be displayed outside of the observability
solution, it's probably not necessary anymore to decouple app definition
and registration like this because it will always be visible no matter
the solution. Once this is the case, the navigation registration can be
moved into the central `observability` plugin, like it's handled with
other apps like infra.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
There is no need to start servers (~1.5 min run time) if there are no
tests matching filters or maybe config itself has all tests skipped.
This PR uses Playwright cli with `--list` flag to quickly validate
playwright config and exit with status code `2` (`1` is reserved for
errors during servers start or test failures). it also useful to know in
advance how many tests were about to run:
case 1: tests found
```
$ node scripts/scout.js run-tests --config x-pack/platform/plugins/private/discover_enhanced/ui_tests/parallel.playwright.config.ts --serverless=security
info scout: Test server configuration saved at /Users/dmle/github/kibana/.scout/servers/local.json
info scout: Validate Playwright config has tests
info scout: Total: 5 tests in 2 files
info Verifying Docker is installed.
│ info Docker version 20.10.14, build a224086349
...
```
case 2: no tests found
```
$ node scripts/scout.js run-tests --config x-pack/solutions/observability/plugins/observability_onboarding/ui_tests/playwright.config.ts --stateful
info scout: Test server configuration saved at /Users/dmle/github/kibana/.scout/servers/local.json
info scout: Validate Playwright config has tests
ERROR scout: No tests found in [x-pack/solutions/observability/plugins/observability_onboarding/ui_tests/playwright.config.ts]
```
## Summary
There's a high likelihood that this causes some unwanted behavior where
the promise is not resolved and the `node` process just exists without
any error.
## Summary
This PR aims at relocating some of the Kibana modules (plugins and
packages) into a new folder structure, according to the _Sustainable
Kibana Architecture_ initiative.
> [!IMPORTANT]
> * We kindly ask you to:
> * Manually fix the errors in the error section below (if there are
any).
> * Search for the `packages[\/\\]` and `plugins[\/\\]` patterns in the
source code (Babel and Eslint config files), and update them
appropriately.
> * Manually review
`.buildkite/scripts/pipelines/pull_request/pipeline.ts` to ensure that
any CI pipeline customizations continue to be correctly applied after
the changed path names
> * Review all of the updated files, specially the `.ts` and `.js` files
listed in the sections below, as some of them contain relative paths
that have been updated.
> * Think of potential impact of the move, including tooling and
configuration files that can be pointing to the relocated modules. E.g.:
> * customised eslint rules
> * docs pointing to source code
> [!NOTE]
> * This PR has been auto-generated.
> * Any manual contributions will be lost if the 'relocate' script is
re-run.
> * Try to obtain the missing reviews / approvals before applying manual
fixes, and/or keep your changes in a .patch / git stash.
> * Please use
[#sustainable_kibana_architecture](https://elastic.slack.com/archives/C07TCKTA22E)
Slack channel for feedback.
Are you trying to rebase this PR to solve merge conflicts? Please follow
the steps describe
[here](https://elastic.slack.com/archives/C07TCKTA22E/p1734019532879269?thread_ts=1734019339.935419&cid=C07TCKTA22E).
#### 3 packages(s) are going to be relocated:
| Id | Target folder |
| -- | ------------- |
| `@kbn/securitysolution-data-table` |
`x-pack/solutions/security/packages/data-table` |
| `@kbn/ecs-data-quality-dashboard` |
`x-pack/solutions/security/packages/ecs-data-quality-dashboard` |
| `@kbn/security-solution-side-nav` |
`x-pack/solutions/security/packages/side-nav` |
<details >
<summary>Updated references</summary>
```
./.i18nrc.json
./package.json
./packages/kbn-ts-projects/config-paths.json
./src/platform/packages/private/kbn-repo-packages/package-map.json
./tsconfig.base.json
./tsconfig.base.type_check.json
./tsconfig.refs.json
./x-pack/solutions/security/packages/data-table/jest.config.js
./x-pack/solutions/security/packages/ecs-data-quality-dashboard/jest.config.js
./x-pack/solutions/security/packages/side-nav/jest.config.js
./yarn.lock
.github/CODEOWNERS
```
</details><details >
<summary>Updated relative paths</summary>
```
x-pack/solutions/security/packages/data-table/jest.config.js:11
x-pack/solutions/security/packages/data-table/tsconfig.json:2
x-pack/solutions/security/packages/ecs-data-quality-dashboard/jest.config.js:24
x-pack/solutions/security/packages/ecs-data-quality-dashboard/tsconfig.json:10
x-pack/solutions/security/packages/ecs-data-quality-dashboard/tsconfig.json:2
x-pack/solutions/security/packages/side-nav/jest.config.js:10
x-pack/solutions/security/packages/side-nav/tsconfig.json:2
```
</details>
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Most of the packages under `/packages` have been relocated in the
context of _Sustainable Kibana Architecture_.
The remaining packages are `devOnly: true`, and they can be grouped as
follows:
1. Packages that are ONLY used from /scripts/
2. Packages that are used from platform and solutions modules (they are
used from tests, cypress tests, storybook configs, ./scripts/ folders
inside some modules, or other non-prod-time logic).
This PR categorises the packages in (2) as `platform/(private|shared)`
to reflect that they are being used from platform and solutions modules
(even though they're used from non-production code).
Next, we're gonna have to decide whether we want to relocate some of
them (1, 2 or both) under a different path.
## Summary
As part of the SKA, some groups of modules have been broken-down and
relocated in different folders, negatively impacting DX. This is the
case for `serverless` modules, that are spread across:
```
- packages/serverless # storybook config
- src/platform/packages/private/serverless
- src/platform/packages/shared/serverless
```
The goal of this PR is to unify them back under
`src/platform/packages/shared/severless`.
In this scenario, I believe the simplification is worth the trade off.
CC @elastic/kibana-management @elastic/appex-sharedux WDYT?
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
As part of the SKA, some groups of modules have been broken-down and
relocated in different folders, negatively impacting DX. This is the
case for `kbn-management` modules, that are spread across:
```
- packages/kbn-management (storybook config)
- src/platform/packages/private/kbn-management
- src/platform/packages/shared/kbn-management
```
The goal of this PR is to unify them back under
`src/platform/packages/shared/kbn-management`.
In this scenario, I believe the simplification is worth the trade off.
CC @elastic/kibana-management WDYT?
Co-authored-by: Matthew Kime <matt@mattki.me>
## Summary
This PR add a method to wait for map to be loaded to replace generic
`renderable.waitForRender()`.
While investigating the recent test failure on CI I found out that for
maps case we can simplify the logic with few facts:
- before start waiting for render to complete, we need to wait for main
container `#maps-plugin` to be in DOM. It takes 2-3 seconds.
- there is always a single div block with `data-render-complete`
attribute, and there is a comment in source code stating `See if the
"data-render-complete" attribute is "true". If so we're done!` which
means we can simply wait for
`div[data-dom-id][data-render-complete="true"]`
6de2ef0e6d/x-pack/platform/plugins/shared/maps/public/connected_components/map_container/map_container.tsx (L103-L116)
`renderable.waitForRender()` is a good waiter, but probably for
dashboard with multiple panels.
## Summary
This PR aims at relocating some of the Kibana modules (plugins and
packages) into a new folder structure, according to the _Sustainable
Kibana Architecture_ initiative.
> [!IMPORTANT]
> * We kindly ask you to:
> * Manually fix the errors in the error section below (if there are
any).
> * Search for the `packages[\/\\]` and `plugins[\/\\]` patterns in the
source code (Babel and Eslint config files), and update them
appropriately.
> * Manually review
`.buildkite/scripts/pipelines/pull_request/pipeline.ts` to ensure that
any CI pipeline customizations continue to be correctly applied after
the changed path names
> * Review all of the updated files, specially the `.ts` and `.js` files
listed in the sections below, as some of them contain relative paths
that have been updated.
> * Think of potential impact of the move, including tooling and
configuration files that can be pointing to the relocated modules. E.g.:
> * customised eslint rules
> * docs pointing to source code
> [!NOTE]
> * This PR has been auto-generated.
> * Any manual contributions will be lost if the 'relocate' script is
re-run.
> * Try to obtain the missing reviews / approvals before applying manual
fixes, and/or keep your changes in a .patch / git stash.
> * Please use
[#sustainable_kibana_architecture](https://elastic.slack.com/archives/C07TCKTA22E)
Slack channel for feedback.
Are you trying to rebase this PR to solve merge conflicts? Please follow
the steps describe
[here](https://elastic.slack.com/archives/C07TCKTA22E/p1734019532879269?thread_ts=1734019339.935419&cid=C07TCKTA22E).
#### 3 packages(s) are going to be relocated:
| Id | Target folder |
| -- | ------------- |
| `@kbn/response-ops-alerts-apis` |
`src/platform/packages/shared/response-ops/alerts-apis` |
| `@kbn/response-ops-alerts-fields-browser` |
`src/platform/packages/shared/response-ops/alerts-fields-browser` |
| `@kbn/response-ops-alerts-table` |
`src/platform/packages/shared/response-ops/alerts-table` |
<details >
<summary>Updated references</summary>
```
./package.json
./packages/kbn-ts-projects/config-paths.json
./src/platform/packages/private/kbn-repo-packages/package-map.json
./src/platform/packages/shared/response-ops/alerts-apis/jest.config.js
./src/platform/packages/shared/response-ops/alerts-fields-browser/jest.config.js
./src/platform/packages/shared/response-ops/alerts-table/jest.config.js
./tsconfig.base.json
./yarn.lock
.github/CODEOWNERS
```
</details><details >
<summary>Updated relative paths</summary>
```
src/platform/packages/shared/response-ops/alerts-apis/jest.config.js:12
src/platform/packages/shared/response-ops/alerts-apis/tsconfig.json:2
src/platform/packages/shared/response-ops/alerts-fields-browser/jest.config.js:12
src/platform/packages/shared/response-ops/alerts-fields-browser/tsconfig.json:2
src/platform/packages/shared/response-ops/alerts-table/jest.config.js:12
src/platform/packages/shared/response-ops/alerts-table/tsconfig.json:2
```
</details>
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Umberto Pepato <umbopepato@users.noreply.github.com>
## Summary
Closes#210340
This PR adds synthtrace clients to scout as a test fixture, so you can
use it in your test to generate data.
The clients added were `apmSynthtraceEsClient`,
`infraSynthtraceEsClient` and `otelSynthtraceEsClient`.
## How to use them in parallel tests
As `synthtrace` ingests data into our indices, and sequential runs would
be the perfect way to introduce flakiness in our tests, there is a
better way to ingest data, using a hook, at the setup phase with
`globalSetup`.
We need to create a `global_setup.ts` file and link it into our
playwright config.
Then we can use something like
```
async function globalSetup(config: FullConfig) {
const data = {
apm: [
opbeans({
from: new Date(start).getTime(),
to: new Date(end).getTime(),
}),
],
infra: [
generateHosts({
from: new Date(start).toISOString(),
to: new Date(end).toISOString(),
}),
],
otel: [
sendotlp({
from: new Date(start).getTime(),
to: new Date(end).getTime(),
}),
],
};
return ingestSynthtraceDataHook(config, data);
}
```
Each key (apm, infra, otel) accepts an array of generators.
## How to use them in sequential tests
> [!WARNING]
> This should not be the standard behaviour, we should embrace
parallelism and use sequential testing when there is no other way.
### apmSynthtraceEsClient
```ts
test.before(
async ({ apmSynthtraceEsClient }) => {
await apmSynthtraceEsClient.index(
opbeans({
from: new Date(start).getTime(),
to: new Date(end).getTime(),
})
);
}
);
```
[opbeans
file](https://github.com/elastic/kibana/blob/main/x-pack/solutions/observability/plugins/apm/ftr_e2e/cypress/fixtures/synthtrace/opbeans.ts)
used in the example.
### otelSynthtraceEsClient
```ts
test.before(
async ({otelSynthtraceEsClient }) => {
await otelSynthtraceEsClient.index(
sendotlp({
from: new Date(start).getTime(),
to: new Date(end).getTime(),
})
);
}
);
```
[sendotlp
file](https://github.com/elastic/kibana/blob/main/x-pack/solutions/observability/plugins/apm/ftr_e2e/cypress/fixtures/synthtrace/sendotlp.ts)
which will create the data.
### infraSynthtraceEsClient
```ts
test.before(
async ({ infraSynthtraceEsClient }) => {
await infraSynthtraceEsClient.index(
generateHosts({
from: new Date(start).toISOString(),
to: new Date(end).toISOString(),
})
);
}
);
```
[generateHosts
file](https://github.com/elastic/kibana/blob/main/x-pack/solutions/observability/plugins/inventory/e2e/cypress/e2e/alert_count/generate_data.ts#L82)
used to generate data.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This PR solves an issue detected in the populate bundle cache plugin
after the webpack v5 migration. On the new version webpack v5 returns a
lot of incomplete paths when we walk over file dependencies or internal
modules. The heuristic logic used previously was faulty so the fixes
turns to use a cached filesystem api instead.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/21030
It was a quick work to do while we don't have many tests yet.
For reviewers: we most likely will review and update the rules to align
better with final test design for Scout before GA. I don't think we have
to deep dive into what rules are missing, but just to make sure I didn't
restrict something important from your perspective.
Rules are described in
https://github.com/playwright-community/eslint-plugin-playwright?tab=readme-ov-file#rules
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
### Summary
- Closes#89741
This PR contains the resulting work of a massive effort that ports our
on top bundler abstraction (called @kbn/optimizer) from Webpack v4 into
Webpack v5. It's essential in terms of long term maintenance since v4
was not receiving updates any longer but will also unblock some new
features that could be beneficial for our future DevEx endeavours.
Next you can find a small list of all the accomplished tasks on this
journey.
### Completed Tasks
- [x] Upgrade dependencies to match the ones on webpack v5
- [x] Fix null-loader usages
- [x] Fix raw-loader usages
- [x] Fix file-loader usages
- [x] Fix url-loader usages
- [x] Fix `@kbn/optimizer-webpack-helpers` to support webpack v5
- [x] Adopt previous webpack v4 polyfill-all strategy with
node-polyfill-webpack-plugin
- [x] Fix theme-loader on @kbn/optimizer
- [x] Migrate configurations and ad-hoc loader options on all webpack
configs from v4 to v5
- [x] Fix @kbn/test jest resolver for file-loader cases
- [x] Migrate public-path loader on UiSharedDeps
- [x] Fix all usages of webpack-merge
- [x] Migrate BundleRemoteModule
- [x] Migrate BundleRemotesPlugin
- [x] Correctly migrate PopulateBundleCachePlugin
- [x] Correctly migrate BundleMetricsPlugin
- [x] Check if the profiling plugins still work (--profile flag)
- [x] Recover if possible the previous webpack v4 cacheGroup chunks
rename to something like `data.plugin.chunk.0.js`
- [x] Run `/ci` and make sure we get our first green CI, otherwise work
on the errors until we do
- [x] Profile and solve bottlenecks until we get a cold build
performance similar to the one we had on webpack v4 (`node
scripts/build_kibana_platform_plugins --no-cache`).
- [x] OpenSSL Legacy Warnings: try to remove `--openssl-legacy-provider
` flags
- [x] Add Webpack to Renovate config
- [x] Explore removing `NodePolyfillPlugin`
([here](https://www.npmjs.com/package/node-polyfill-webpack-plugin)) and
add each polyfill needed individually per each webpack config to check
if we get smaller bundles. If we do it's better to go with the case by
case need approach instead of deploying a bunch of polyfills with
NodePolyfillPlugin. As another alternative, create a custom smaller
plugin with only the union of all needed polyfills.
- [x] Evaluate if we want to touch the resolutions on mainFields and
conditionNames
- [x] Understand why `@import 'src/core/public/mixins'` does not work
anymore (not a problem, we should use relative paths anyway but we want
to track why it changed from v4 to v5)
- [x] BUG: Child compilers are having errors hidden and/or changed from
error to warning
- [x] Fix license check for
[Artistic-2.0](https://spdx.org/licenses/Artistic-2.0.html) is the
license for
[domain-browser](https://github.com/bevry/domain-browser?tab=License-1-ov-file).
This package is a dependency of
[NodePolyfillPlugin](https://www.npmjs.com/package/node-polyfill-webpack-plugin).
Artistic 2.0 license is [classified as
yellow](https://github.com/elastic/open-source/blob/main/elastic-product-policy.md#yellow-list)
and should only be used for dev dependencies.
- [x] Make sure `resourceQuery: { not: /raw/ }` is not necessary on
other webpack configs like storybook one
- [x] Find what is being wrongly removed by usedExports optimization;
hint: I believe it is identifying a lot of exports inside the sync entry
of plugins as unused exports and removing them. Then `__kbnBootstrap__`
can't be found
- [x] Rebalance @kbn/optimizer pickMaxWorkerCount
- [x] Re-open the issue to fix sass-warnings
[#190345](https://github.com/elastic/kibana/issues/190345) or downgrade
sass-loader to v10
- [x] Remove previous esm no parse rules
- [x] Confirm esm support is working
- [x] Confirm console override is needed
- [x] Confirm react prod builds on ui shared deps for distributable
- [x] Remove customization for
[xyflow](https://github.com/xyflow/xyflow) from webpack configs
- [x] Clean all the code
- [x] Make sure collected metrics from stats are still aligned with what
we were collecting before; also verify if the modules used for optimizer
caches etc are well generated (@kbn/node-libs-browser)
- [x] Fix watch performance
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Brad White <brad.white@elastic.co>
Addresses #206436
- Replace styled-components with @emotion/react and @emotion/styled
across multiple components
- Remove `@kbn/ui-theme` instances
- Update test cases to use @emotion/jest matchers for toHaveStyleRule to
work properly
- Adjust styles to use `css` from `@emotion/react`
- Remove unused imports and updated snapshots
https://github.com/user-attachments/assets/48b9ab41-23fd-4c00-ace9-c0cbe58c91f7
## Summary
This PR adds a test config category to the scout reporting. This allows
us to distinguish between UI and API FTR tests.
A new property `testConfigCategory` has been added to all FTR configs
that don't already inherit it from a higher level config.
## Summary
This PR turns the AlertsTable into a standalone component, making it
independent from the `TriggersActionsUI` plugin.
#### Removes the alerts table registry
All configuration is now managed through the AlertsTable component
props. Shared configurations are handled by giving consumers the ability
to directly provide alerts table wrapper components (see for example the
`renderAlertsTable` prop of `getCases`).
#### Moves the alerts table to dedicated package(s)
Following the feature-driven structure we're introducing for ResponseOps
(alerting) client-side packages:
- `@kbn/response-ops-alerts-table`
- `@kbn/response-ops-alerts-apis`
- `@kbn/response-ops-alerts-fields-browser`
#### Initial work on improving composition and organization
- Reorganizes the table code into a by-entity-type folder structure
(`components/`, `hooks/`, ...)
- Simplifies some components and breaks into smaller units when possible
## To verify
For consumers of the alerts table:
- Check that all your tables have the same behavior as before (columns,
sort, row actions, bulk actions, etc.)
- Check that your "shared" tables (i.e. cases alerts view in O11y and
Security) have the expected configuration and behavior
> [!WARNING]
> This PR moves a lot of files. Git might not always recognize the
correct delete/add file pairs. If you see weird diffs feel free to reach
out for help!
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
### Identify risks
| Risk | Description | Severity | Mitigation |
|---|---|---|---|
| Table misconfigurations | Some table configurations might slightly
differ from the previous AlertsTableRegistry-backed version | Low |
Quick fix |
## References
Closes#195180
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Christos Nasikas <xristosnasikas@gmail.com>
## Summary
- [x] Removed use of color calculating function `transparentize()`
- [x] Begin removal of JSON tokens when possible in favor of colors
tokens
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR is probably the final PR that makes the changes to support EUI
Borealis. It focuses on removing all the usage of EUI Json tokens.
You will notice different approaches while removing the tokens:
- for some cases, the changes were done using `css from
'@emotions/react'` as the components using the tokens were already using
`euiTheme` or adding it was straightforward and required the minimal
amount of changes
- for some cases, where the css changes were pretty involved, a hook was
created to be able to import the styles and apply them in the components
- finally for other cases, esepcially if the styled components were
extracted in a different file and were used within many others, I
decided to create reusable components. This allowed to not change all
the files impacted and limit the number of files modified in this PR.
Feel free to comment on any of the approaches and suggest better
options!
https://github.com/elastic/kibana/issues/201889
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
`@kbn/scout-oblt` is a test library that extends `@kbn/scout` with test
helpers specifically designed to test `Observability` applications in
Kibana. All Oblt plugins should only import from `@kbn/scout-oblt`
Its primary goal is to simplify the test development experience for
teams working on `Observability` plugins by providing custom Playwright
fixtures, page objects, and utilities tailored for Observability-related
testing scenarios.
Contributing:
- when Fixture/Page Object is sharable across all Solutions and Platform
(`fleetApi` fixture), it should be added in `@kbn/scout`
- when Fixture/Page Object is Oblt-specific but is shared across tests
under the multiple plugins (`OnboardingHome` page), it should be added
in `@kbn/scout-oblt`
- when Fixture/Page Object is only used in a single plugin (`onboarding`
internal APIs ?), it should be added in this plugin.
I also re-worked existing tests with few ideas in mind:
- Scout is **e2e testing tool** and should target primary e2e test
scenarios; We have _API integration tests_ to test multiple short
scenarios for APIs behavior (response, status code) and _jest/React
testing library_ to test components in isolation (elements rendering,
fields validation). Doing all the testing with e2e tool like Playwright
will dramatically affect cost efficiency and stability of tests, but
also slows overall CI execution and PRs delivery. The goal is to follow
testing pyramid and keep in mind its principles.
- We on purpose spin up new browser context for each `test` block to
make sure our **tests are independent**. Having too many short `test`
blocks in the file significantly slows down the execution: every block
triggers browser context, saml authentication, adding/removing Fleet
integrations (each call up to 2 seconds) and other beforeEach/afterEach
hooks. Real browser-based testing is expensive. It is not about putting
every step into 1 `test` block, but also not a Jest unit-test-style
design. When it is possible to group similar actions on the same page
and if it is a part of the same user flow - we should do it. It also
doesn't bring the testing value repeating the same UI steps multiple
times in different scenarios. _Our CI costs are critical to cut when it
is possible_
- Avoid **nesting describe** blocks: it complicates test readability and
also complicates for CI bot to properly skip the failing block (it will
skip the top level one). We encourage **Scout parallel test execution**
based on running test spec files in multiple workers, not the `test`
blocks within the same file. Having too many `test` blocks in the same
file will be slowly run in the single thread and in case of flakiness,
it means Team lose more test coverage than they probably expect.
Before (**59** test blocks - **8-8.5 min** per distro):
<img width="1709" alt="Screenshot 2025-02-08 at 18 01 40"
src="https://github.com/user-attachments/assets/5fd65a1c-85f9-4594-9dae-3f8e99a005ab"
/>
After (**15** test blocks - **3.5-4 min** per distro):
<img width="1578" alt="Screenshot 2025-02-10 at 18 14 42"
src="https://github.com/user-attachments/assets/6846898f-7dd2-4f6b-8bc5-d06741b0b120"
/>
For reviewers: updated tests are possible to run in 2 parallel workers
against the same Kibana/ES instance and run time is dropping to **2.5-3
min** 🚀 . It is up to UX-Logs team to decide if you want to keep
parallel run (new tests can be added either to parallel or sequential
run)
<img width="1578" alt="Screenshot 2025-02-11 at 12 14 30"
src="https://github.com/user-attachments/assets/e94113f2-d7f1-470e-a6d5-cb5154d99c41"
/>
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR migrates the last routes with `access:<privilege>` tags used in
route definitions to new security configuration.
Please refer to the documentation for more information: [Authorization
API](https://docs.elastic.dev/kibana-dev-docs/key-concepts/security-api-authorization)
### **Before Migration:**
Access control tags were defined in the `options` object of the route:
```ts
router.get({
path: '/api/path',
options: {
tags: ['access:<privilege_1>', 'access:<privilege_2>'],
},
...
}, handler);
```
### **After Migration:**
Tags have been replaced with the more robust
`security.authz.requiredPrivileges` field under `security`:
```ts
router.get({
path: '/api/path',
security: {
authz: {
requiredPrivileges: ['<privilege_1>', '<privilege_2>'],
},
},
...
}, handler);
```
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Part of https://github.com/elastic/kibana/issues/194171
PR removes all async imports run during uptime plugin setup and start
methods. This causes page load bundle size to increase and accurately
reflect its true size.
### Test instructions
* Start kibana locally
* Open network tab in browser
* Open home page. Verify `uptime.chunk` are not loaded. The screen shots
show the behavior in main where `uptime.chunk` are loaded on home page
<img width="600" alt="Screenshot 2025-02-05 at 9 06 56 AM"
src="https://github.com/user-attachments/assets/14218b85-3814-4e3c-9c04-bd73cf6c4dbd"
/>
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
This PR removes the `isDraggable` prop throughout Security Solution.
Unless I'm mistaken, this property isn't necessary anymore, as we do not
use those draggable elements anymore. From what I could see, we had its
value set to `false` everywhere.
This lead to a lot of files impacted, but most of them have only a
couple of lines changed. In some files though, removing the
`isDraggable` prop allowed to remove more code than became obsolete.
**No UI changes should have been introduced in this PR!**
### What this PR does
- removes `isDraggable` everywhere
- performs the extra small cleanup when obvious
- updates all corresponding unit e2e and tests
### What this PR does
- rename files or component names to limit the already extensive impact
of the code change
Resolves#209261
## Summary
Removes the code used to render Logs Explorer. This does not result in
any functional changes.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
When we run Scout tests in parallel, we call SAML authentication in
parallel too and since by default `.security-profile-8` index does not
exist, we periodically getting 503 response:
```
proc [kibana] [2025-01-29T11:13:10.420+01:00][ERROR][plugins.security.user-profile]
Failed to activate user profile: {"error":{"root_cause":[{"type":"unavailable_shards_exception","reason":
"at least one search shard for the index [.security-profile-8] is unavailable"}],
"type":"unavailable_shards_exception","reason":"at least one search shard
for the index [.security-profile-8] is unavailable"},"status":503}. {"service":{"node":
{"roles":["background_tasks","ui"]}}}
```
The solution is to retry the SAML callback assuming that index will be
created and the issue will be solved.
We agreed with Kibana-Security to retry only **5xx** errors, because for
**4xx** we most likely have to start the authentication from the start.
For reviews: it is not 100% reproducible, so I added unit tests to
verify the retry logic is working only for 5xx requests. Please let me
know if I miss something
Retry was verified locally, you might be seeing this logs output:
```
proc [kibana] [2025-01-30T18:40:41.348+01:00][ERROR][plugins.security.user-profile] Failed to activate user profile:
{"error":{"root_cause":[{"type":"unavailable_shards_exception","reason":"at least one search shard for the index
[.security-profile-8] is unavailable"}],"type":"unavailable_shards_exception","reason":"at least one search shard
for the index [.security-profile-8] is unavailable"},"status":503}. {"service":{"node":{"roles":["background_tasks","ui"]}}}
proc [kibana] [2025-01-30T18:40:41.349+01:00][ERROR][plugins.security.authentication] Login attempt with "saml"
provider failed due to unexpected error: {"error":{"root_cause":[{"type":"unavailable_shards_exception","reason":
"at least one search shard for the index [.security-profile-8] is unavailable"}],"type":"unavailable_shards_exception",
"reason":"at least one search shard for the index [.security-profile-8] is unavailable"},"status":503}
{"service":{"node":{"roles":["background_tasks","ui"]}}}
proc [kibana] [2025-01-30T18:40:41.349+01:00][ERROR][http] 500 Server Error {"http":{"response":{"status_code":500},"request":{"method":"post","path":"/api/security/saml/callback"}},"error":
{"message":"unavailable_shards_exception\n\tRoot causes:\n\t\tunavailable_shards_exception: at least one
search shard for the index [.security-profile-8] is
ERROR [scout] SAML callback failed: expected 302, got 500
Waiting 939 ms before the next attempt
proc [playwright]
info [o.e.c.r.a.AllocationService] [scout] current.health="GREEN" message="Cluster health status changed
from [YELLOW] to [GREEN] (reason: [shards started [[.security-profile-8][0]]])."
previous.health="YELLOW" reason="shards started [[.security-profile-8][0]]"
```
To reproduce:
```
node scripts/scout.js run-tests --stateful --config x-pack/platform/plugins/private/discover_enhanced/ui_tests/parallel.playwright.config.ts
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
* Categorise and move `@kbn/timelines-plugin` as _platform/shared_,
target location: `x-pack/platform/plugins/shared/timelines`.
This helps reduce the scope of the illegal dependencies from `osquery`
plugin towards _security/private_ code.
cc @tomsonpl
* Simplify path and rename `@kbn/observability-alerting-rule-utils`
(platform/shared):
```
# Before
@kbn/observability-alerting-rule-utils
x-pack/platform/packages/shared/observability/alerting_rule_utils/
# After
@kbn/alerting-rule-utils
x-pack/platform/packages/shared/alerting_rule_utils/
```
* Simplify path and rename `@kbn/observability-logs-overview`
(platform/shared):
```
# Before
@kbn/observability-logs-overview
x-pack/platform/packages/shared/observability/logs_overview/
# After
@kbn/logs-overview
x-pack/platform/packages/shared/logs_overview/
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Sets the traceparent for Playwright, so the trace from the test runner
includes the trace events from the browser and Kibana server.
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
This PR makes few changes to scout reporter:
Recently I found out that Playwright load configuration file _multiple
times_:
- first time after you run `npx playwright test -c ...`
- on every worker start
log before:
```
[main][~/github/kibana]$ npx playwright test --config x-pack/platform/plugins/shared/maps/ui_tests/playwright.config.ts
[createPlaywrightConfig] called with runId: 18f35f735a10155c
Running 1 test using 1 worker
[createPlaywrightConfig] called with runId: 2633b4e4c20afa15
[chromium] › full_screen_mode.spec.ts:28:9 › Maps › Full screen mode @svlSecurity @svlOblt @svlSearch @ess
```
With our current logic unique `runId` will be generated on each
configuration load, meaning for parallel run we will report failures in
different directories instead of the same one.
Playwright doesn't expose any unique identifier for the run, so we have
do something similar described in
https://github.com/microsoft/playwright/issues/28941#issuecomment-1888008102
log after fix:
```
[main][~/github/kibana]$ npx playwright test --config x-pack/platform/plugins/shared/maps/ui_tests/playwright.config.ts
[createPlaywrightConfig] called with runId: 310a576f32d3b8a5
Running 1 test using 1 worker
[createPlaywrightConfig] called with runId: 310a576f32d3b8a5
[chromium] › full_screen_mode.spec.ts:28:9 › Maps › Full screen mode @svlSecurity @svlOblt @svlSearch @ess
```
We also had a chat with @dolaru and agreed that Scout reporters to be
disabled for local test run. Few reasons:
- Scout custom reporting targets CI execution: events-based reporter was
already disabled
- Failed test reporter purpose is to provide html boilerplate to be
annotated in pipeline build
- When you run tests with IDE playwright plugin it provides its own
reporter / history, should be enough.
### Background
In some scenarios we need to perform some setup at bootstrap time, this
setup could affect following scenarios.
Take for example
[failed_logs](https://github.com/elastic/kibana/blob/main/packages/kbn-apm-synthtrace/src/scenarios/failed_logs.ts)
scenario where we create a pipeline that will do some checks in
`log.level` property, if we try to run an scenario after that one we
will enter into some issues.
### Changes
This PR aims to introduce a `teardown` setup for scenarios where we
could undo the changes done at `bootstrap` time.
## Summary
This PR is a followup to #205990, which removed references to all of the
deprecated/renamed EUI vars in preparation for 9.0. Here, we address
some of the non-critical tech debt related to the EUI refresh, namely
the [removal of static EUI
tokens](https://github.com/elastic/kibana/issues/199715#json-tokens)
from our codebase.
I made every attempt not to change any styles in this PR, except to
simplify CSS to produce an equivalent design. A common example of this
was removing a static `margin` or `padding` declaration referencing
`euiThemeVars.size*`, and swapping it with an equivalent `gutterSize`
prop on the `EuiFlexGroup` container, or with an `align-self` or other
equivalent flexbox directive.
## Screenshots of Areas Affected
The majority of changes here involved the Exception List/Item pages.
I've attached screenshots of their current layout for comparison/review:
<details>
<summary>
<h3>Rule Exceptions Tab</h3>
</summary>
<kbd>
<h3>Before</h3>
<img width="1395" alt="Rule exceptions tab - before"
src="https://github.com/user-attachments/assets/db7a5487-7df3-4a5f-b88e-90ab34784970"
/>
</kbd>
<kbd>
<h3>After</h3>
<img width="1421" alt="Rule exceptions tab - after"
src="https://github.com/user-attachments/assets/77cbdefc-cbec-4b9e-8436-197f2f2f6677"
/>
</kbd>
</details>
<details>
<summary>
<h3>Shared Exception Lists</h3>
</summary>
<kbd>
<img width="1517" alt="Shared Exception Lists"
src="https://github.com/user-attachments/assets/5448dd65-60f7-470c-bf7a-7af75bb914fa"
/>
</kbd>
</details>
<details>
<summary>
<h3>Shared Exception List Details</h3>
</summary>
<kbd>
<img width="1517" alt="Shared Exception List Details"
src="https://github.com/user-attachments/assets/6e03fef6-af58-40bb-98c5-188651a584cc"
/>
</kbd>
</details>
<details>
<summary>
<h3>Threshold Input</h3>
</summary>
<kbd>
<img width="1046" alt="Threshold Input"
src="https://github.com/user-attachments/assets/28738857-6bdf-404f-a790-a9f4e66ff27a"
/>
</kbd>
</details>
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
