## Summary
* Address https://github.com/elastic/kibana/issues/217145 - Put in place
a check to ensure we're upgrading from Kibana 8.18.0 or newer.
* Address https://github.com/elastic/kibana/issues/220521 - New attempt
at removing the `switchToModelVersionAt` property, inspired on
https://github.com/elastic/kibana/pull/219029.
The previous attempt caused a regression: index meta information started
storing _modelVersions_ that were older than the previously stored ones,
which were defaulting to 10.0.0 for SO types that did NOT define
`modelVersions`.
This was due to the removal of the applyTypeDefaults, which was ensuring
all SOs had the `switchToModelVersionAt` property set.
This flag was then used by
`src/core/packages/saved-objects/base-server-internal/src/model_version/version_map.ts`
to determine whether to use `modelVersions` or the legacy `migrations`
property in order to determine the latest model version for a given
type.
When removing the `switchToModelVersionAt` flag (and its default
backfill), the logic started defaulting to the latest `migrations`
version for those SO types that were not defining any `modelVersion`,
resulting in older versions that those stored in the SO indices. This
caused incident https://elasticco.atlassian.net/browse/INC-3818.
This regression has been shipped in 9.0.0 (the PR was
[backported](https://github.com/elastic/kibana/pull/219329)), so in top
of the cleanup, we now need to address
https://github.com/elastic/kibana/issues/220521 to ensure a smooth
transition _OnPrem => Serverless_.
---------
Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR applies **lossless compression** to all SVG and JPG/PNG assets
across Kibana using:
- [`svgo`](https://github.com/svg/svgo) — for optimizing SVGs
- [`image-optimize`](https://www.npmjs.com/package/image-optimize) — for
JPG/PNG compression
‼️**Please scroll to ''Unknown metric groups" accordion to see what's
the gain for your code.**
<img width="542" alt="Screenshot 2025-06-18 at 13 24 20"
src="https://github.com/user-attachments/assets/191afb28-44fc-4551-9026-756a8385c66a"
/>
The goal is to reduce asset size and improve load performance without
compromising visual quality.
This PR achieves a **23 MB** reduction in asset size across all images
bundled in Kibana’s running code—meaning these compressed images
directly impact what ships in Kibana.
Some assets get bundled into chunks due to our bundling strategy but
might not actually be requested at runtime.
Additionally, I ran the same optimization script on the docs assets as a
harmless extra step, but those savings aren’t included in the 23 MB
total.
---
## Why
While working on Emotion rewrites, I noticed some SVGs seemed
unnecessarily heavy. That led to a broader investigation into our image
assets, and it turns out we’re not consistently optimizing them during
development or build.
---
## Notes
- Visual fidelity of optimized assets has been manually verified — no
visible differences
- The optimization is **lossless**, meaning no quality degradation
- Some assets (like large background images) could benefit further from
**lossy compression**
---
## Follow-ups / Ideas
1. **Automate compression in the dev/build pipeline**
- e.g. add `svgo` as a pre-commit or CI step for SVGs
2. **Improve CI reporting**
- Currently, bundle size diffs for images are hidden under "Unknown
metric groups" in the GitHub CI comment. We may want to make these more
visible.
-
3. **Audit large assets manually** — apply lossy compression where
appropriate
4. **Avoid redundant image loading**
- e.g. background images on the login page are loaded again on the space
selector page since they’re bundled twice. I’m working on a separate PR
to address that.
## Snippets I used to apply the compression
```
# Find SVG files
find . -type f -iname "*.svg" \
-not -path "*/node_modules/*" \
-not -path "*/functional/*" > svg-files.txt
# Compress SVGs
while IFS= read -r file; do
svgo "$file"
done < svg-files.txt
```
This snippet has been used for png and jpg, but the example below is for
png:
```
# Find PNG files
find . -type f -iname "*.png \
-not -path "*/node_modules/*" \
-not -path "*/functional/*" > png-files.txt
# Compress PNGs
while IFS= read -r file; do
image-optimize -f jpg "$file"
done < png-files.txt
```
Enhances the performance metrics documentation by explaining how
to use the `meta.description` field for contextualizing render time
events and how to track subsequent page or section loads using
`onPageRefreshStart`.
- Added a section describing the default meaning of render time and how
to provide a custom `description` in the `meta` field of `onPageReady`
for more precise event context.
- Documented the use of `onPageRefreshStart` to distinguish between
initial loads and subsequent refreshes, clarifying that
`meta.isInitialLoad` is set to `false` for refreshes and `true` by
default.
- Included code examples and sample indexed event structures for both
features.
## Summary
Part of https://github.com/elastic/kibana/issues/201807
**Strategy**:
This PR assumes that `switchToModelVersionAt` has done it’s job and all
SO type owners now use `modelVersions` instead of `migrations`.
It takes the approach of trusting that this is fine™
**Changes in this PR:**
- `switchToModelVersionAt` removed from all registered saved objects, so
their mapping hash had to be updated.
- Implements `globalSwitchToModelVersionAt` directly in `version_map`
- Updates logic to prioritize `modelVersions` over `migrations` in
`getLatestMappingsModelVersion`, that previously relied on
`switchToModelVersionAt` being set
- Updates unit tests to match the refactored logic
- Updates snapshots
- Adapts SO types registered in the **cases**, **SLO** and **encrypted
saved objects** plugins.
- Updates docs
**Risks**:
- Plugin developers disregard the deprecated status of `migrations` and
introduce new versions, which will not be BWC. Saved object type hash
changes will notify the core team for a code owner review. The core team
needs to investigate the related changes and provide feedback.
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Introduces a new `security_solution/gen_ai_evals.yml` BuildKite pipeline
for automatically running our Assistant and Attack Discovery evaluation
suites weekly.
### To Run Locally:
Ensure you are authenticated with vault for LLM + LangSmith creds:
> See [internal
docs](https://github.com/elastic/infra/blob/master/docs/vault/README.md#login-with-your-okta)
for setup/login instructions.
Fetch Connectors and LangSmith creds:
> [!NOTE]
> In discussion with @elastic/kibana-operations it was preferred to use
the ci-prod secrets vault, so we cannot self-manage the secrets. To test
this locally though, you can grab the secrets and follow the
instructions in this [paste
bin](https://p.elstc.co/paste/q7k+zYOc#PN0kasw11u2J0XWC2Ls5PMNWreKzKTpgWA1wtsPzeH+).
```
cd x-pack/test/security_solution_api_integration
node scripts/genai/vault/retrieve_secrets.js
```
Navigate to api integration directory, load the env vars, and start
server:
```
cd x-pack/test/security_solution_api_integration
export KIBANA_SECURITY_TESTING_AI_CONNECTORS=$(base64 -w 0 < scripts/genai/vault/connector_config.json) && export KIBANA_SECURITY_TESTING_LANGSMITH_KEY=$(base64 -w 0 < scripts/genai/vault/langsmith_key.txt)
yarn genai_evals:server:ess
```
Then in another terminal, load vars and run the tests:
```
cd x-pack/test/security_solution_api_integration
export KIBANA_SECURITY_TESTING_AI_CONNECTORS=$(base64 -w 0 < scripts/genai/vault/connector_config.json) && export KIBANA_SECURITY_TESTING_LANGSMITH_KEY=$(base64 -w 0 < scripts/genai/vault/langsmith_key.txt)
yarn genai_evals🏃ess
```
### To manually run on BuildKite:
Navigate to
[BuildKite](https://buildkite.com/elastic?filter=ftr-security-solution-gen-ai-evaluations)
and run `ftr-security-solution-gen-ai-evaluations` pipeline.
### To manually run on BuildKite for specific PR:
In `.buildkite/ftr_security_stateful_configs.yml`, temporarily move the
`genai/evaluations/trial_license_complete_tier/configs/ess.config.ts`
line down to the `enabled` section. Will see if we can do this without
requiring a commit. @elastic/kibana-operations is it possible to set a
buildkite env var that can be read in FTR tests when a specific GitHub
label is added to the PR? I.e. can I create a `SecurityGenAI:Run Evals`
label that when added will run this suite as part of the build?
> [!NOTE]
> Currently the connectors secrets only include `gpt-4o` and
`gpt-4o-mini`. Waiting on finalized list w/ credentials from @jamesspi
and @peluja1012 and then we can have ops update using the scripts
included in this PR.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Patryk Kopycinski <patryk.kopycinski@elastic.co>
## Summary
- Closeselastic/kibana-operations#245
- Convert FIPS base image from UBI to `chainguard-base-fips`
- Add FIPS base image updates to Renovate
- Adjust naming scheme for FIPS image from `kibana-ubi-fips` to
`kibana-fips`
- Adds new image flavor `kibana-cloud-fips`
- Adds support for `ci:build-cloud-fips-image` label
- Move Cloud image building to its own step instead of being part of
`Build Kibana Distribution` step so it will be triggered when the build
is reused and the `build` step is skipped.
Currently Kibana forwards `query_range_secs` and `query_offset_secs` to
mark the selected time range when reporting TTFMP event. This format
caused some challenges to identify `from`, `to` date offsets in
visualizations.
To simplify, the PR renames and sends the three fields explicitly:
- `query_from_offset_secs` offset to `0` (now), with -ve for past and
+ve for future dates
- `query_to_offset_secs` offset to `0` (now), with -ve for past and +ve
for future dates
- `query_range_secs` same as previously sent
_This approach is followed after a discussion, and based on the
[gist](https://gist.github.com/andrewvc/1f04a57a336d768e4ec5ff2eff06ba54)
excerpt:_
```
Earliest date -> QueryFrom
Newest date -> QueryTo
Duration -> QueryRange
```
### Indexing
These fields then should be mapped in the EBT indexer to ingest in the
top level of the document, eventually removing the need to create
runtime fields in data views for visualizations.
Also, runtime fields in data views should be updated to reflect this
change. For backward compatibility, the runtime fields can cater both
the old and new field names conditionally.
### Testing
- Ensure that the TTFMP events are correctly reporting the date ranges.
### Example
## Summary
The `/packages` folder at the root of the Kibana repository used to
contain a lot of packages.
In the context of SKA, they have been gradually moved to various
locations:
* `src/platform/packages`
* `x-pack/platform/packages`
* `src/core/packages`
Currently, only `devOnly: true` packages are left in this folder. This
comprises libraries for CLI scripts as well as testing utilities.
With this PR, we are moving ~half of these packages under
`src/platform/packages/(private|shared)/`.
In particular, we are moving those packages that are being used from
platform and/or solutions.
Since they are `"devOnly": true`, this means they are ONLY used from
tests, cypress tests, storybook configs, ./scripts/ folders inside some
modules, or other non-prod-time logic. Nonetheless, they are effectively
referenced from platform and/or solutions code, hence I decided they
should be placed under `platform` folders.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Adding documentation to help teams diagnose FIPS pipeline errors in
test.
Linked to FIPS compliant OpenSSL/Node setup docs
---------
Co-authored-by: Jeramy Soucy <jeramy.soucy@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Relates to https://github.com/elastic/kibana/issues/181995
This PR updates the examples to include availability information and
another description.
Co-authored-by: Jean-Louis Leysens <jeanlouis.leysens@elastic.co>
Saved objects declared as `hidden` can only be accessed with a client
that explicitly includes hidden types.
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
---------
Co-authored-by: Alejandro Fernández Haro <afharo@gmail.com>
* Remove duplicated “File service” entry from nav
* Move Screenshotting to main Tutorials section in nav
* Add “Updating Puppeteer and Chromium” to nav as a sub-item of
screenshotting
* Move files for Screenshotting/Chromium out of the SharedUX space to
`dev_docs/tutorials/screenshotting`
## Summary
It’s common request for Dev teams to run specific journeys on a PR to
compare performance metrics against the `main` branch. These requests
usually focus on a particular area, such as the Dashboard or Discover
app.
To streamline the process, this PR groups relevant journeys into
categories that can be triggered through an environment variable. For
example, setting `JOURNEYS_GROUP=dashboard` will execute only the three
dashboard-specific journeys, which are (usually) sufficient for
evaluating the performance impact of code changes within the Dashboard
app.
Current Process for Triggering Performance Builds:
- Create a new kibana-single-user-performance
[build](https://buildkite.com/elastic/kibana-single-user-performance#new)
- Provide the following arguments:
Branch: `refs/pull/<PR_number>/head`
Under Options, set the environment variable:
`JOURNEYS_GROUP=<group_name>`
Currently supported journey groups:
- kibanaStartAndLoad
- crud
- dashboard
- discover
- maps
- ml
[Build example
](https://buildkite.com/elastic/kibana-single-user-performance/builds/14427)
Each group focuses on a specific set of journeys tied to its respective
area in Kibana, allowing for more targeted performance testing. Since
running group takes ~5-10 min on bare metal worker, it should not delay
the regular (every 3h) runs against `main` branch
test locally with `node scripts/run_performance.js --group <group_name>`
## Summary
Added documentation explaining how SO migrations on serverless work
## Preview
<img width="741" alt="Screenshot 2024-03-22 at 16 15 13"
src="2217c01f-8447-4f22-a782-a07ff221aa42">
## Summary
Moving synthtrace clients init inside kbn-journeys:
esArchiver does not always solve the issue with data generation. We
already have afew journeys using Synthtrace instead and expect more to
come.
In order to simplify the process of creating new journeys, this PR moves
Synthtrace client initialisation into kbn-journey package and exposes a
way to define client type, generator function & its input arguments:
```
import { Journey, SynthtraceOptions } from '@kbn/journeys';
import { subj } from '@kbn/test-subj-selector';
import { generateApmData } from '../synthtrace_data/apm_data';
export const journey = new Journey({
synthtrace: {
type: 'apm',
generator: generateApmData,
options: {
from: new Date(Date.now() - 1000 * 60 * 15),
to: new Date(Date.now() + 1000 * 60 * 15),
},
},
})
```
PR also needs review from teams who use Synthtrace to understand if the implementation is matching expectations.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
The source code classifier we currently have was incorrectly classifying
e2e journey files as `non-package` instead of `tests or mocks` as it was
not using the name standards we used for FTR files.
We could have created a `functional-tests` package for the performance
folder (which is what we want to do in the future) but because we don't
have the feature to create ownerless packages it would not be easy to
find a given owner for that folder.
As such I'm just opting for a second solution which is applying the same
name standards to this journeys folder as we have for FTR and changing a
little the classifier to recognise it.
This should fix the problem found at
https://github.com/elastic/kibana/pull/178017.
Co-authored-by: Alex Szabo <alex.szabo@elastic.co>
## Summary
Closes elastic/kibana-operations/issues/24
This adds a second flavor of UBI image (`kibana-ubi-fips`) which has a
FIPS compliant version of OpenSSL compiled and linked to Node. Using the
label `ci:build-docker-fips` will create the image in CI and push to the
registry.
The FIPS image start the Kibana NodeJS process using the FIPS compliant
OpenSSL version. Kibana will start in this state but crash during
runtime because there are many code changes required for it to be FIPS
compliant, including `node_module` usage. I attempted numerous ways to
load other OpenSSL providers alongside the FIPS provider, but it always
led to Kibana crashing on invalid algorithm usage.
---------
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
I had to change `waitForRender` since `page.waitForFunction` tries to
run a script on page and it is not working due to CSP settings on Cloud.
Instead of injecting a script, we use a classical API to find
elements/attributes in the DOM.
Since `PUT /internal/core/_settings` is merged in 8.11.0, journeys run
on Cloud with on-fly labels update is supported starting deployments
8.11.0+. I added error message for 404 code just in case someone runs it
on earlier version.
`many_fields_discover` journey was update since on Cloud the data view
used by scenario is not selected by default.
How it works:
Create a deployment with QAF and re-configure it for journey run:
```
export EC_DEPLOYMENT_NAME=my-run-8.11
qaf elastic-cloud deployments create --stack-version 8.11.0-SNAPSHOT --environment staging --region gcp-us-central1
qaf elastic-cloud deployments configure-for-performance-journeys
```
Run any journey, e.g. many_fields_discover
```
TEST_CLOUD=1 TEST_ES_URL=https://username:pswd@es_url:443 TEST_KIBANA_URL=https://username:pswd@kibana-ur_url node scripts/functional_test_runner --config x-pack/performance/journeys/many_fields_discover.ts
```
You should see a log about labels being updated:
```
Updating telemetry & APM labels: {"testJobId":"local-a3272047-6724-44d1-9a61-5c79781b06a1","testBuildId":"local-d8edbace-f441-4ba9-ac83-5909be3acf2a","journeyName":"many_fields_discover","ftrConfig":"x-pack/performance/journeys/many_fields_discover.ts"}
```
And then able to find APM logs for the journey in
[Ops](https://kibana-ops-e2e-perf.kb.us-central1.gcp.cloud.es.io:9243/app/apm/services?comparisonEnabled=true&environment=ENVIRONMENT_ALL&kuery=labels.testJobId%20%3A%20%22local-d79a878c-cc7a-423b-b884-c9b6b1a8d781%22&latencyAggregationType=avg&offset=1d&rangeFrom=now-24h%2Fh&rangeTo=now&serviceGroup=&transactionType=request)
cluster
## Summary
Expands on the HTTP versioning tutorial with more information about
`internal` vs `public` endpoints.
## Screenshots
<img width="875" alt="Screenshot 2023-06-01 at 15 52 18"
src="f8f790a0-31ec-4123-89db-b1f01aa17845">
<img width="923" alt="Screenshot 2023-06-01 at 15 52 33"
src="f97ad89b-face-4176-a814-8e17ecd3d2a5">
* Add deferred migrations parameter.
* Update outdated documents query to take into account deferred migrations.
* Update outdated documents query to take into account the core migration version.
* Update read operations in the saved objects repository to perform deferred migrations.
