## Summary
This PR introduces the first building blocks for the [Entity Analytics
Privileged
Monitoring](https://github.com/elastic/security-team/issues/9971).
We follow the approach used in the Entity Store and add a new "Engine",
which consists of the following components:
* Public API
* INIT and HEALTH routes
* Kibana task
* Privilege Monitoring Data Client
* Engine Saved Object
* API key manager
* Related storage indices
* Feature Flag: `privilegeMonitoringEnabled` set to `false` by default.
* API integration test configuration
* only tests that the health endpoint is available
* Auditing and Telemetry
## Testing steps
1. Make sure to add `privilegeMonitoringEnabled` to your
`kibana.dev.yaml`
2. In devtools, ensure the API is working with `GET
kbn:/api/entity_analytics/monitoring/privileges/health`
3. Start the engine with: `POST
kbn:/api/entity_analytics/monitoring/engine/init`
4. Look for `DEBUG` logs mentioning the
`entity_analytics:monitoring:privileges:engine` task
---------
Co-authored-by: CAWilson94 <charlotte.wilson@elastic.co>
Co-authored-by: Charlotte Alexandra Wilson <CAWilson94@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR adds the structure for workchat FTR tests and adds a few initial
tests as an example.
### Details about initially added tests
New test directories:
- `x-pack/test_serverless/api_integration/test_suites/chat`
- load a few common tests (that run on all project types)
- run `platform` security tests (taken over from `search` project type)
- `x-pack/test_serverless/functional/services/svl_chat_navigation.ts`
- load the `home page` common test
- run a simple navigation test, using the `svlChatNavigation` service
that has been introduced as an example
Note that these tests mostly serve as examples to prove things are
actually running and will have to be adjusted / removed / extended over
time. The purpose of this PR is NOT to add proper test coverage.
Closes#213469
---------
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>
## Summary
Splitting long running config:
`x-pack/test/functional/apps/saved_query_management/config.ts` **~57
min** into
- x-pack/test/functional/apps/saved_query_management/config.ts 35m
- x-pack/test/functional/apps/saved_query_management/config.v2.ts 25m
17s
ideally we need to split both even more, but I will leave it for the
later (probably Data-Discovery Team have some ideas how to re-org it?)
## Summary
Splitting the following config:
-
x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts
**~61 min**
by moving `ai_assistant`, `synthetics` and `streams` tests in its own
configs
-
x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.ai_assistant.serverless.config.ts
~11m 30s
-
x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.synthetics.serverless.config.ts
~21m 30s
-
x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.streams.serverless.config.ts
~21m 43s
original config with less tests:
-
x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts
~17 min
## Summary
Adjusts the Docker tag for the Wolfi FIPS image from `kibana-fips` to
`kibana-wolfi-fips` to avoid confusion in the future. The other products
use `<product>-fips` naming for released Cloud artifacts but our
artifact is `kibana-cloud-fips`.
### Considerations
This changeset could be further reaching, but unsure if it's necessary
and would like other opinions. If we want to change it now is the time
while adoption is low. For example, we're using `--skip-docker-fips` in
build scripts or GH label `ci:build-docker-fips`. We could align these
better, adding `wolfi` but don't think it is necessary.
## Summary
* Removes manually maintained OAS for Core's `_export` and `_import`
Saved Object APIs
* seeks to remove some code duplication and move our OAS descriptions
closer to the code definitions.
## Summary
```
TimeoutError: page.waitForSelector: Timeout 10000ms exceeded.
Call log:
- waiting for locator('[data-test-subj="globalLoadingIndicator-hidden"]')
at Object.waitForSelector (src/platform/packages/shared/kbn-scout/src/playwright/fixtures/test/scout_page/single_thread.ts:47:27)
at Page.waitForLoadingIndicatorHidden (src/platform/packages/shared/kbn-scout/src/playwright/fixtures/test/scout_page/single_thread.ts:91:27)
at x-pack/solutions/security/plugins/security_solution/ui_tests/parallel_tests/flyout/alert_details_url_sync.spec.ts:40:16
```
Failure rate before changes: 60%
<img width="487" alt="image"
src="https://github.com/user-attachments/assets/9331f973-9337-48cf-9131-c3acbf611d9c"
/>
For some reason specifically in serverless Security project loading
indicator is not hidden on page reload after 10 seconds.
To unblock Teams this PR changes tests to wait for
`detectionsAlertsPage` to be visible instead. The follow-up is to set
loading status directly in Alerting Table component and wait for
`alertsTable-loaded` data-test-subj in UI tests before interacting with
table.
Failure rate with changes: 0%
<img width="1592" alt="Screenshot 2025-04-04 at 17 08 32"
src="https://github.com/user-attachments/assets/c955ed94-bc92-4328-a3b5-0194806d31b1"
/>
## Summary
Currently, if you'd like to test something on Kibana's VM image, you'd
have to build a VM image to -qa, then rewrite all references to
`elastic-images-qa` for the PR jobs; once done with testing, we'd undo
the changes to `elastic-images-prod`.
This is a helpful tool for us to test with WIP VM images, we'd be able
to add a label to the PR, and it would automatically grab the QA images,
without any temporary commits.
Jobs in https://buildkite.com/elastic/kibana-pull-request/builds/289599
have ran with an elastic-qa image. ✅
## Summary
- Closeselastic/kibana-operations#245
- Convert FIPS base image from UBI to `chainguard-base-fips`
- Add FIPS base image updates to Renovate
- Adjust naming scheme for FIPS image from `kibana-ubi-fips` to
`kibana-fips`
- Adds new image flavor `kibana-cloud-fips`
- Adds support for `ci:build-cloud-fips-image` label
- Move Cloud image building to its own step instead of being part of
`Build Kibana Distribution` step so it will be triggered when the build
is reused and the `build` step is skipped.
## Summary
According to:
https://buildkite.com/elastic/kibana-on-merge/builds/65027#0195ca29-b10a-4e20-b00f-c4fbe43689fa
```
Annotate test failures error Request failed with status code 404 AxiosError: Request failed with status code 404
--
| at settle (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1742853500882456889/elastic/kibana-on-merge/kibana/.buildkite/node_modules/axios/lib/core/settle.js:19:12)
...
| at async /opt/buildkite-agent/builds/bk-agent-prod-gcp-1742853500882456889/elastic/kibana-on-merge/kibana/.buildkite/scripts/lifecycle/annotate_test_failures.ts:14:5
| HTTP Error 404/Not Found (https://api.buildkite.com/v2/organizations/elastic/pipelines/kibana-on-merge/builds/65027/artifacts?page=2&per_page=100) { message: 'Not Found' }
```
This points to the client collecting all artifacts through traversing
the `next` links from Buildkite's API responses. It appears, Axios is
not happy about these absolute paths, even if the origin is the same.
This PR adjusts the next link parsing to relativize compared to a base
url.
## Summary
While annotating test failures, we're seeing increased amount of errors
like this:
```
2025-03-21 13:52:32 INFO Artifact uploads completed successfully
--
| Annotate test failures error Request failed with status code 404
| HTTP Error Response Status 404
| HTTP Error Response Body { message: 'Not Found' }
| user command error: exit status 10
```
It would be nicer to show a bit more from the error to help debugging.
## Summary
A [recent PR](https://github.com/elastic/kibana/pull/212558/files) (3
weeks ago) migrated docs under `/docs` folder, from `.asciidoc` to
`.md`.
Unfortunately, some of these docs (well, at least one) were generated
automatically using a script:
`node scripts/build_plugin_list_docs.js` was updating the
`docs/developer/plugin-list.asciidoc`
As a result of the migration:
The CI pipeline step fails, and the list of plugins is no longer
updated.
Besides:
* The https://www.elastic.co/guide/en/kibana/current/plugin-list.html is
currently broken.
* This page is (or should be) built from
https://github.com/elastic/kibana/blob/main/docs/extend/plugin-list.md,
which is also broken.
They are broken cause some plugins' descriptions use the pipe `|`
character, which breaks the Markdown layout.
This PR updates the logic that generates the plugin-list, switching from
`.asciidoc` to `.md`, and escaping `\n` and `|` characters to allow for
better descriptions.
## Summary
Extracts `collectEnvFromLabels` to a separate module, so it can be used
in the flaky test runner. With this, the label `ci:use-chrome-beta` will
be passed along to the flaky test runner, allowing for flaky testing on
chrome beta.
Other labels we treat as modifiers for PR behavior through setting env
variables should also be added to this set of mapping.
This reverts commit d8f6bd694b.
## Summary
Since this upgrade, we're getting 404 on failed test annotation.
Reverting this while we figure out what's causing it.
## Summary
Skips the basic license cases list view test since it is expecting the
cases list to not to be present for a basic/essentials tier license, but
the FIPS pipeline always runs with a platinum license override.
## Summary
Closes https://github.com/elastic/kibana/issues/210298
In this PR we are adding the initial structure for the
@kibana/scout-security package, note that this is not ready to be used
and any new test using this package, is not going to be executed as part
of the regular pipelines, meaning, you are not going to add coverage to
the application.
@kibana/scout-security package is a test package that extends @kbn/scout
with test helpers specifically designed to test Security Solution
functionalities in Kibana. All tests under Security plugins should only
import from @kbn/scout-security, not from @kbn/scout.
This PR is a POC to start testing development by providing custom
Playwright fixtures, page objects, and utilities tailored for
Security-related testing scenarios.
Things to follow-up:
- CustomQueryRule interface is already declared in
`x-pack/solutions/security/plugins/security_solution/common/api/detection_engine/model/rule_schema/rule_schemas.gen.ts`
as `QueryRuleCreateProps`
- DETECTION_ENGINE_RULES_URL and DETECTION_ENGINE_RULES_BULK_ACTION are
already declared in `@kbn/security-solution-plugin/common/constants`
It would be great if all of that is extracted from the plugin to a
package so it can be reused instead of having to duplicate the code.
Until the package is not ready to be used and has not been introduced to
the different teams, appex-qa and myself will be the owners of it to
make sure that best practices are followed
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
## Summary
This PR establishes the foundation for executing API tests in the new
`search_ai_lake` tier, following the existing API integration test
structure and guidelines.
## Adding a New Test
To add a new test, follow these guidelines:
- Inside the `AI4DSOC` folder, create subfolders representing different
AI4DSOC functionalities.
- Each subfolder should be owned by an area team or the developers
actively working on it.
- The functionality folder must include a `search_ai_lake_tier`
subfolder.
- The `search_ai_lake_tier` subfolder should contain a `configs`
directory with a `serverless.config.ts` file that imports
`createTestConfig` from `config.base.ai4dsoc`.
- Add the test inside the `search_ai_lake_tier` subfolder.
- Ensure the test has the `@serverless` label and uses
`supertestWithoutAuth` instead of `supertest`, as `supertest` provides
basic authentication, whereas serverless environments require API key
authentication. See the `dummy_test.ts` for reference.
- The `search_ai_lake_tier` folder should have an `index.ts` file
referencing the tests to be executed, as demonstrated in this PR.
- Update the
`x-pack/test/security_solution_api_integration/package.json` file with
the necessary scripts to enable test execution locally.
- When adding a new `serverless.config.ts` file, ensure it is included
in `.buildkite/ftr_security_serverless_configs.yml`. Otherwise, the new
test(s) will not be executed as part of the PR process.
## Running Tests Locally
Execute the tests using the following Yarn scripts from
`x-pack/test/security_solution_api_integration`:
1. Start the server with the required configuration:
```sh
yarn ai4dsoc_cases:server:serverless
```
2. Run the tests using the started server:
```sh
yarn ai4dsoc_cases🏃serverless
```
## Key Considerations
- `Supertest` should not be used, as it provides basic authentication.
Instead, use supertestWithoutAuth for API key authentication.
- All tests must include the `@serverless` label.
- MKI is not yet supported for test execution.
- Temporary Ownership: The Security Engineering Productivity team will
initially own the AI4DSOC testing folder to ensure proper structure and
best practices. Once teams are familiar with the workflow, this
ownership will be removed.
## Security Engineering Productivity Code Ownership Responsibilities
The Security Engineering Productivity team should ensure:
- All tests are placed inside a functionality-specific subfolder.
- Each functionality subfolder has designated code owners.
- Tests include the `@serverless` label.
- `Supertest` is not used.
- The correct configuration is applied.
- Scripts are added to enable local execution.
- New configurations are added to
`.buildkite/ftr_security_serverless_configs.yml`.
## Follow-Up tasks
- Remove the existing dummy test.
- Integrate tests into the periodic pipeline.
- Add tests to the Kibana QA quality gate.
## Summary
The intent is to have a centralised place to store the list of Kibana
solutions and serverless project types.
To that end, this PR creates a `@kbn/projects-solutions-groups` package.
It also adds the new solution type `'chat'`.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR establishes the baseline to execute Cypress tests in the new
`search_ai_lake` tier.
## Changes Introduced
- All tests under
`x-pack/test/security_solution_cypress/cypress/e2e/ai4dsoc` will be
executed using the new tier by default.
- These tests will run as part of the PR process within the `Serverless
AI4DSOC - Security Solution Cypress Tests` execution.
## Adding a New Test
To add a new test, follow these guidelines:
- Read the
[README](x-pack/test/security_solution_cypress/cypress/e2e/ai4dsoc/README.md).
- Inside the `AI4DSOC` folder, we should have different subfolders
representing the various AI4DSOC functionalities.
- Each subfolder should have ownership by either an area team or the
developers actively working on it.
- Make sure that any functionality you want to be tested in the new tier
is added inside the `AI4DSOC` folder; otherwise, that functionality will
be tested using the complete tier.
## Running Tests Locally
Run the tests with the following Yarn scripts from
`x-pack/test/security_solution_cypress`:
```sh
yarn cypress:open:ai4dsoc:serverless
```
Opens the Cypress UI with all tests in the `e2e/ai4dsoc` directory. This
also runs a mocked serverless environment using the `ai_soc` product
line and `search_ai_lake` tier by default.
```sh
yarn cypress:run:ai4dsoc:serverless
```
Runs all tests tagged as @serverless in the e2e/ai4dsoc directory in
headless mode using the ai_soc product line and search_ai_lake tier by
default.
## Key Considerations
- All tests must have the `@serverless` tag to be executed as part of
the PR process.
- MKI is not yet supported for test execution.
- The AI4DSOC Cypress tests will be executed each time there is a change
in one of its
[dependencies](https://github.com/elastic/kibana/blob/main/.buildkite/scripts/pipelines/pull_request/pipeline.ts).
- All tests are executed by default using the `platform_engineer` role.
- Temporary Ownership: The Security Engineering Productivity team will
own the entire AI4DSOC testing folder initially to ensure structure and
best practices. Once all teams understand the workflow, this ownership
will be removed.
- Execution Time: If test execution in a PR takes more than 45 minutes,
parallelism should be increased in the new
`.buildkite/pipelines/pull_request/security_solution/ai4dsoc.yml` file.
## Security Engineering Productivity Codeownership Responsibilities
The Security Engineering Productivity team should ensure:
- Best practices are followed.
- All tests are placed inside a functionality subfolder.
- Each functionality subfolder has designated code owners.
- Tests include the `@serverless` label.
- The execution of AI4DSOC tests does not exceed 45 minutes.
## Follow-Up Tasks
- Remove the dummy test (@tomsonpl feel free to delete it when you need
to add new tests to the navigation).
- Integrate tests into the periodic pipeline.
- Add tests to the Kibana QA quality gate.
- Update the README with MKI instructions once tests are added to the
periodic pipeline and Kibana QA quality gate.
- Clarify which roles will be used for the AI4DSOC effort and update the
tests accordingly.
## Summary
More teams are adding Scout tests in their plugins, often as a PoC and
not stable yet for continuous execution.
We don't want to block it, but need a way to manage the scope of Scout
pipeline and be able to disable it quickly to unblock the Scout
development.
Since Scout is in active development and we need it to be simple and
quick as possible (we can iterate and improve later), we agreed with
Robert to disable tests by plugin:
```
ui_tests:
enabled:
- apm
- discover_enhanced
- maps
- observability_onboarding
disabled:
- *skipped_plugin*
```
When scout configuration is added to the new plugin, it will require to
update `.buildkite/scout_ci_config.yml` that is owned by `appex-qa`
team. If there is no intention to run Scout tests on CI, plugin name
should be added under `disabled` section.
**How to test locally:**
- Scout tests were added in `observability_onboarding` plugin, pipeline
will throw error
modify locally `.buildkite/scout_ci_config.yml`
```
ui_tests:
enabled:
- apm
- discover_enhanced
- maps
disabled:
```
run `node scripts/scout discover-playwright-configs --validate --save`
```
ERROR The following plugins are not registered in Scout CI config '.buildkite/scout_ci_config.yml'
- observability_onboarding
```
~~On CI annotation will be added to clarify the failure:~~
we decided to move validation to "Quick Checks", no need to annotate.
<img width="1583" alt="image"
src="https://github.com/user-attachments/assets/ed6b5778-74cb-4473-8218-b96239aab067"
/>
- `observability_onboarding` plugin is disabled, pipeline won't include
it (excluded in `scout_playwright_configs.json`)
modify locally `.buildkite/scout_ci_config.yml`
```
ui_tests:
enabled:
- apm
- discover_enhanced
- maps
disabled:
- observability_onboarding
```
run `node scripts/scout discover-playwright-configs --validate --save`
```
warn The following plugins are disabled in '.buildkite/scout_ci_config.yml' and will be excluded from CI run
- observability_onboarding
info Found Playwright config files in '4' plugins.
Saved '3' plugins to '/Users/dmle/github/kibana/.scout/test_configs/scout_playwright_configs.json'
```
This PR updates the ES|QL grammars (lexer and parser) to match the
latest version in Elasticsearch.
---------
Co-authored-by: drewdaemon <drew.tate@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
## Summary
Fix broken tests !!
These got broken due to changes on alerts overview page, i am also
expanding the scope to run on all observability plugin changes !!
Add streams API to documentation as an experimental feature
<img width="2555" alt="Screenshot 2025-03-07 at 11 44 54"
src="https://github.com/user-attachments/assets/f54e5e6e-0c20-4bad-9cff-27747d0f76e2"
/>
There are a couple of changes in here:
* Split streams API in internal and public and mark the public parts as
experimental
* Add the public parts to the Kibana documentation
* Add description and summary
* Adjust the server repository wrapper to pass through summary and
description
# To test
* Generate OAS bundle: `node scripts/capture_oas_snapshot --include-path
/api/streams --update`
* Apply overlays `cd oas_docs && make api-docs`
* Make sure bump.sh is installed (`npm install -g bump-cli`)
* Run for preview: `cd oas_docs && bump preview output/kibana.yaml`
# Open questions
* Does the split into public and internal make sense?
* Is it a problem if this is visible in the user-facing documentation
page before we actually release streams? Or would it be OK if the API is
marked as experimental? (mostly a question for @LucaWintergerst )
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Trying to fix pipeline failure due to not enough disk space:
```
| 2025-03-12 10:47:33 UTC | Copying cached snapshots from /opt/buildkite-agent/.es-snapshot-cache/cache to .es/cache
| 2025-03-12 10:47:48 UTC | cp: error writing '.es/cache/elasticsearch-9.0.0-SNAPSHOT-linux-x86_64.tar.gz': No space left on device
| 2025-03-12 10:47:48 UTC | cp: error writing '.es/cache/elasticsearch-9.0.0-SNAPSHOT-linux-x86_64.tar.gz.meta': No space left on device
| 2025-03-12 10:47:48 UTC | cp: error writing '.es/cache/elasticsearch-9.1.0-SNAPSHOT-linux-x86_64.tar.gz': No space left on device
| 2025-03-12 10:47:48 UTC | cp: error writing '.es/cache/elasticsearch-9.1.0-SNAPSHOT-linux-x86_64.tar.gz.meta': No space left on device
```
## Summary
Closes - https://github.com/elastic/kibana/issues/166679
## What's included ?
- The PR adds a feature in Logs View of Observability (to start with) to
hide the regular pagination toolbar from the footer and show Load More
only when the user has scrolled to the bottom of the page.
- The table would always load the items in batches of default set 500
- This PR also add 2 helper functions `useThrottleFn` and
`useDebounceFn`. Current React help library which KIbana uses called
-`react-use` does not have these and we cannot use Lodash variant of
these. We need such hooks which are React safe. Hence added these 2
## What's pending ?
- [x] Unit tests for the 2 new helper React hooks
- [x] Unit tests for data table footer component
- [x] Unit tests for Profile Resolution
- [x] Functional Serverless Tests
- [x] Functional Stateful Tests
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Davis McPhee <davismcphee@hotmail.com>
Co-authored-by: Felix Stürmer <weltenwort@users.noreply.github.com>
Co-authored-by: Davis McPhee <davis.mcphee@elastic.co>
## Summary
related to https://github.com/elastic/kibana/pull/211797
Fixing bootstrap failure in
https://buildkite.com/elastic/kibana-on-merge-unsupported-ftrs/builds/34167
```
yarn install and bootstrap, attempt 2
2025-03-11 18:13:27 UTC yarn run v1.22.22
2025-03-11 18:13:27 UTC $ node scripts/kbn bootstrap --force-install
2025-03-11 18:13:27 UTC Kibana should not be run as root. Use --allow-root to continue.
2025-03-11 18:13:27 UTC error Command failed with exit code 1.
2025-03-11 18:13:27 UTC info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
2025-03-11 18:13:28 UTC 🚨 Error: The command exited with status 1
```
**Addresses:** https://github.com/elastic/kibana/issues/180267
## Summary
This PR enables `prebuiltRulesCustomizationEnabled` feature flag.
## Details
Besides simply enabling `prebuiltRulesCustomizationEnabled` feature flag the following required changes were done
- failed tests due enabling the FF were fixed
- FF setting was removed from test configurations (integrations and Cypress tests)
- FF logic was removed from the codebase. Disabling the FF would require roll back test changes as well. So just in case we have to disable the FF it's simpler to roll back the PR's commit.
## Summary
closes https://github.com/elastic/kibana/issues/211592
This PR improves the way we run scout tests by discovering all the
plugins that have the scout tests and run tests in multiple workers:
<img width="1586" alt="image"
src="https://github.com/user-attachments/assets/4936ab50-fefb-470c-af3a-21263b58143f"
/>
How it works:
1. Run search script to find _all existing_ scout playwright config
files across kibana repo
2. Save results into `.scout/scout_playwright_configs.json` file, that
will be used as source to run configs in individual jobs per plugin.
Upload it as BK artifact.
3. Spin up job for each plugin mentioned in
`scout_playwright_configs.json`
4. In each individual job use `scout_playwright_configs.json` and get
configs for specific plugin, use worker with 8 vcpus where tests are run
in parallel (`usesParallelWorkers` prop)
While running configs 1 by 1 collect command exit code with the
following rules:
- configs run passed => exit code `0` , final status remains `0`
- config has no tests => exit code `2`, put config name into
`configsWithoutTests` group to push BK annotation later, change exit
status to `0` - we accept configs without tests at current stage
- config run fails => exit code `1`, final status changed to `1` and job
will fail in the end; put config name into `failedConfigs` group to push
BK annotation later
<img width="1564" alt="Screenshot 2025-02-21 at 14 34 16"
src="https://github.com/user-attachments/assets/06e9298d-466c-46bb-8e85-3d691a40850a"
/>
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
When VM image rebuild is triggered after ES promotion, only the cache
warmup should be built.
This PR also separates the daily full build to a daily base + cache
build (in case ES promotions are failing for some reason, we should
still have a daily cache refresh).
Requires: https://github.com/elastic/ci-agent-images/pull/1295
With this, we'd run a daily base image build and cache build (~40m +
25m) + cache warmups for every promotion (~4x 25m) instead of a full
build and promotion per build (~4x 55m). Ultimately not that much of a
gain 🤷 (4*55=220m => 40+5x25=165m)
