**Addresses**: https://github.com/elastic/kibana/issues/184428
## Summary
This PR adds scripts for automatic bundling of Endpoint Management API OpenAPI specs as a part of PR pipeline. Corresponding result bundles are automatically committed to the Security Solution plugin `x-pack/plugins/security_solution` in the `docs/openapi/ess/` and `docs/openapi/serverless` folders (similar to https://github.com/elastic/kibana/pull/186384).
**Addresses**: https://github.com/elastic/kibana/issues/184428
## Summary
This PR adds scripts for automatic bundling of Timeline API OpenAPI
specs as a part of PR pipeline. Corresponding result bundles are
automatically committed to the Security Solution plugin
`x-pack/plugins/security_solution` in the `docs/openapi/ess/` and
`docs/openapi/serverless` folders (similar to
https://github.com/elastic/kibana/pull/186384).
---------
Co-authored-by: Jan Monschke <jan.monschke@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
**Addresses**: https://github.com/elastic/kibana/issues/184428
## Summary
This PR adds scripts for automatic bundling of Osquery API OpenAPI specs as a part of PR pipeline. Corresponding result bundles are automatically committed to the Osquery plugin `x-pack/plugins/osquery` in the `docs/openapi/ess` and `docs/openapi/serverless` folders (similar to https://github.com/elastic/kibana/pull/186384).
**Addresses**: https://github.com/elastic/kibana/issues/184428
## Summary
This PR adds scripts for automatic bundling of AI Assistant API OpenAPI specs as a part of PR pipeline. Corresponding result bundles are automatically committed to the Assistant Common package `x-pack/packages/kbn-elastic-assistant-common` in the `docs/openapi/ess/` and `docs/openapi/serverless` folders (similar to https://github.com/elastic/kibana/pull/186384).
**Addresses**: https://github.com/elastic/kibana/issues/184428
## Summary
This PR adds scripts for automatic bundling of Entity Analytics API OpenAPI specs as a part of PR pipeline. Corresponding result bundles are automatically committed to the Security Solution plugin `x-pack/plugins/osquery` in the `docs/openapi/ess` and `docs/openapi/serverless` folders (similar to https://github.com/elastic/kibana/pull/186384).
Closes https://github.com/elastic/kibana/issues/186631
Closes https://github.com/elastic/kibana-operations/issues/151
Adds a daily pipeline for running our jest and integration tests against
a Node.js distribution with pointer compression enabled. This is enabled
by setting the environment variable
`CI_FORCE_NODE_POINTER_COMPRESSION=true`
I would prefer a cleaner implementation, but I'm not seeing a way around
it without changing our defaults globally. Open to ideas. We have to
update three downloads:
1) base node.js install, for jest
2) build node.js install, for integration tests
3) bazel workspace install, for dependencies
https://buildkite.com/elastic/kibana-pointer-compression/builds/6
---------
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
**Addresses**: https://github.com/elastic/kibana/issues/184428
## Summary
This PR adds scripts for automatic bundling of Exceptions API OpenAPI specs as a part of PR pipeline. Corresponding resulting bundles are automatically committed in the Lists common package `kbn-securitysolution-exceptions-common` in the `docs/openapi/ess/` and `docs/openapi/serverless` folders (similar to https://github.com/elastic/kibana/pull/186384).
**Addresses**: https://github.com/elastic/kibana/issues/184428
## Summary
This PR adds scripts for automatic bundling of Lists API OpenAPI specs as a part of PR pipeline. Corresponding resulting bundles are automatically committed in the Lists common package `kbn-securitysolution-lists-common` in the `docs/openapi/ess/` and `docs/openapi/serverless` folders (similar to https://github.com/elastic/kibana/pull/186384).
Update i18n tools after the main packages upgrade. This upgrade makes
use of formatJS tooling instead of fully implementing the parsers
ourselves. It also changes our custom AST parsing from babel to the
typescript compiler.
- [x] i18n exrtract
- [x] i18n check
- [x] i18n integrate
- [x] add test cases for formatjs runner
- [x] Make sure all CLI flags are handled properly
- [x] Update tooling readme
Closes https://github.com/elastic/kibana/issues/180616
Closes https://github.com/elastic/kibana/issues/187703
### Note to reviewers
Teams outside operations and core are probably requested to review
because the `i18n_check` fixed malformed i18n messages in your plugins.
Please check and approve :elasticheart:
The upload build artifacts step matches on a glob, which now resolves to
two distributions, breaking future commands. This swaps find for the
specific distribution.
This PR removes the usage of the native module version of `re2` and
replaces it with a js port called `re2js`.
It also ends our usage of native node modules in production and it
removes the task from the build as well. Further steps will be taken
along our strategy to avoid future usages of native node modules in prod
environments.
## Summary
Closes#188272
A check was added to in #181187 which detects if the environment has
FIPS enabled NodeJS, but Kibana is not setup properly. This adds the
Kibana setting for FIPS in CI and the Docker image. Note there are still
license issues on some tests due to #181187 as well, but this will be
handled in another PR.
## Goal
We'd like to introduce a way to run pipelines that have a dependency on
the currently active branch set (managed in
[versions.json](./versions.json)).
With this, we'd like to migrate over the `es-forward` pipelines
(currently:
[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-15-forward-compatibility),
and
[this](https://buildkite.com/elastic/kibana-7-dot-17-es-8-dot-14-forward-compatibility))
to the new buildkite infra.
## Summary
This PR introduces a new pipeline:
https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs
(through
[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml)).
The purpose of this new pipeline is to take the name of a "pipelineSet"
that refers to a pipeline, and based on the `versions.json` file, work
out what are the branches on which the referred pipeline should be
triggered.
### Example: `Trigger ES forward compatibility tests`
- a scheduled run on
[kibana-trigger-version-dependent-jobs](https://buildkite.com/elastic/kibana-trigger-version-dependent-jobs)
with the env var `TRIGGER_PIPELINE_SET=es-forward` runs
- the pipeline implementation for
`kibana-trigger-version-dependent-jobs` works out (looking at
`versions.json`), that the `es-forward` set should trigger
https://buildkite.com/elastic/kibana-es-forward (doesn't exist prior to
the PR) for (7.17+8.14) and (7.17+8.15)
- the pipeline implementation uploads two trigger steps, running
https://buildkite.com/elastic/kibana-es-forward in two instances with
the relevant parameterization.
Since the trigger parameters are derived from the `versions.json` file,
if we move on and close `8.14`, and open up `8.16`, this will follow,
without having to update the pipeline resources or schedules.
## Changes
- 2 pipelines created:
[trigger-version-dependent-jobs.yml](.buildkite/pipeline-resource-definitions/trigger-version-dependent-jobs.yml),
[kibana-es-forward.yml](.buildkite/pipeline-resource-definitions/kibana-es-forward.yml)
- [x] add kibana-es-forward.yml
- implementation for `trigger-version-dependent-jobs` added
- branch configuration removed from pipelines (kibana-artifacts-staging,
kibana-artifacts-snapshot, kibana-artifacts-trigger)
- added a script for checking RREs validity (moved a few files)
## Verification
I've used the migration staging pipeline (*) to run this:
-
https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/130
- Env: `TRIGGER_PIPELINE_SET="artifacts-trigger"`
- Result:
[(success):](https://buildkite.com/elastic/kibana-artifacts-trigger/builds/10806)
it triggered for 8.14 only (as expected)
-
https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/131
- Env: `TRIGGER_PIPELINE_SET="es-forward"`
- Result: (success): it generated 2 trigger steps, but since the
es-forward pipeline doesn't exist, the upload step failed
-
https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/132
- Env: `TRIGGER_PIPELINE_SET="artifacts-snapshot"`
- Result: (success): it triggered jobs for all 3 open branches
(main/8.14/7.17)
-
https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/134
- Env: `TRIGGER_PIPELINE_SET="artifacts-staging"`
- Result: (success): it triggered 8.14 / 7.14, but not for main
(*note: this migration staging pipeline will come in handy even after
the migration, to stage newly created pipelines without creating the
resource up-front)
## Summary
These were used for testing the migration from the kibana-buildkite
infra to the elastic-wide buildkite infra. Now we're done with most of
the migration, we can clean these up.
## Summary
- Removes SSH info to avoid confusion since we cannot SSH into agents on
the new infra
- Removes old agent metrics and logs links because they are in a
different cluster and the new links are in an annotation
Currently, if a container image already exists for a project the script
exits early with a failure. In cases where CI is re-triggered, this can
be inconvenient and require an upstream merge to rebuild.
Instead of exiting early, this skips the image build on pull requests.
## Summary
When we're seeing errors in FTR or on the serverless verification
pipeline, we have difficulty connecting back what version of
ES-Serverless is behind the tag `:latest`.
With a recent addition to the ES Serverless docker image, this info is
now contained in labels of the image.
This PR highlights this info in the verification pipeline, as well as
the FTR output from `kbn-es`.
- Serverless verification pipeline:
https://buildkite.com/elastic/kibana-elasticsearch-serverless-verify-and-promote/builds/1454
- FTR:
- removes parallelism: 2 from step definition. The test suites are not
sharded.
- Updates the path used to trigger a test run. The previous path is out
of date.
This adds the use of the `-f` flag to `curl` commands so that the
`deploy.sh` script will fail if an HTTP response is not 2xx.
This will prevent, for example, us from thinking that an update of a
project worked when instead the API responded `405` and nothing happened.
Related to https://github.com/elastic/kibana/pull/186543.
## Summary
- Closes https://github.com/elastic/kibana-operations/issues/100
- Utilizes FIPS agent from elastic/ci-agent-images#686
- Adds dynamic agent selection during PR pipeline upload
- FIPS agents can be used with `FTR_ENABLE_FIPS_AGENT` env variable or
`ci:enable-fips-agent` label
- Removes agent image config from individual steps in favor of image
config for the whole pipeline.
- Steps can still override this config by adding `image`, `imageProject`
etc
- Adds a conditional assertion to `Check` CI step which validates that
FIPS is working properly
### Testing
- [Pipeline run using FIPS
agents](https://buildkite.com/elastic/kibana-pull-request/builds/215332)
- Failures are expected and this possibly ran with flaky tests
# Pull Request Overview
This pull request (PR):
1. Enables Product Types for FTR API Integration Test Suites in
Serverless MKI:
- Previously, the test suites ignored product types entirely. With this
PR, scripts to run the tests have been relocated to
x-pack/test/security_solution_api_integration/scripts.
- Users can now run tests from the API Integration tests directory for
security solutions by using the command:
TARGET_SCRIPT={script_from_package_json} node
./scripts/mki_start_api_ftr_execution. This will execute the following
steps:
1. Create a security serverless project, respecting the product types
specified in the serverless configuration found in the config folder of
the relevant test suite.
2. Reset credentials.
3. Wait for Elasticsearch (ES) and Kibana to be ready and available.
4. Execute the tests.
5. Delete the project upon completion.
2. Adds Proxy Services Organizations to .ftr Role Users Files:
- This PR updates the .ftr role_users files to include all proxy
services organizations, ensuring they have the necessary permissions for
the tests.
# Implementation Details
Product Types
- Previous Setup:
- A project was created and handed over to the test suite to run the API
tests against, without considering product types.
- Changes Introduced:
- The script execution for tests has moved to
x-pack/test/security_solution_api_integration/scripts.
- Tests can be initiated using the command from the API Integration
tests folder where package.json exists:
```
TARGET_SCRIPT={script_from_package_json} node
./scripts/mki_start_api_ftr_execution
```
- The mki_start_api_ftr_execution script performs several steps to run
the tests, including creating a security serverless project with
specified product types, resetting credentials, ensuring ES and Kibana
readiness, executing tests, and cleaning up the project afterward.
- The script reads extra configuration (currently only product types are
supported) in the specific format as is, from the following file :
[api_configs.json](https://github.com/elastic/kibana/pull/184309/files#diff-1122baffe7ff843b1f486cee95468bed5851a9a4934be747f540bd42dc9a07daR2).
The key for the JSON file is the name of the script in
[package.json](https://github.com/elastic/kibana/pull/184309/files#diff-c6af1c81947b3a77bed431c688c7ad38c8969bd52e1c3ea92d643f09d422eb61R296)
- If a specific configuration is not required and the default complete
project is needed for the test to run, the key and configuration in the
`api_configs.json` file can be ommitted.
# Summary
This PR enhances the flexibility and functionality of the FTR API
integration test suites for serverless MKI by incorporating product type
considerations and updating the role_users configuration to include
proxy services organizations.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
With the migration to the shared buildkite infra, we've also switched to
using the ci-prod vault (https://vault-ci-prod.elastic.dev) for all
CI-related secrets. We found it reasonable then, to also switch the
storage of the credentials for the deployments there. It's since been
proven unnecessary, even confusing for developers, as they might not be
adequately set up for accessing the two vaults. We've also learned, that
both of these vault instances are here to stay, so there's no push to
migrate everything to the ci-prod instance.
So, this PR switches back to using the legacy vault in all cases for
storing deployment keys, as it fits better with the developers' daily
secret handling duties.
Also, adds a cleanup part to the purge routine.
- [x] extract vault read / write to a parametric shell script, because
the typescript invocations to vault won't have an easy access to the
`set_in_legacy_vault`
**Addresses:** https://github.com/elastic/kibana/issues/184428
## Summary
This PR adds scripts for automatic bundling Detections API OpenAPI spec as a part of PR pipeline. Corresponding resulting bundles are automatically committed to `x-pack/plugins/security_solution/docs/openapi/ess/` and `x-pack/plugins/security_solution/docs/openapi/serverless` folders.
## Summary
Only adds build_project.yml if there's not already a deploy_project.yml
added through labels
## Context
Based on the labeling logic, if we had `ci:build-serverless-image` and
`ci:project-deploy-*`, the steps for building the docker image would be
added twice.
See: https://elastic.slack.com/archives/C0D8P2XK5/p1718268306523329
