## Summary
Bump Cypress-related dependencies to the latest versions and update
`renovate.json` to do it automatically in the future
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Gloria Hornero <gloria.hornero@elastic.co>
## Summary
Let's automate E2E against Serverless
Changelog:
- updated certs to include additional dns names we are using for testing
locally, `host.docker.internal`, `es01`
- updated certs generation README to include changes related to
`openssl@3`
- added new certs for Fleet server
- added fleet-server service token
- added support for `ca_trusted_fingerprint` in fleet preconfig
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Tomasz Ciecierski <ciecierskitomek@gmail.com>
Co-authored-by: Tomasz Ciecierski <tomasz.ciecierski@elastic.co>
Co-authored-by: Kevin Logan <kevin.logan@elastic.co>
## Summary
This is hopefully the last batch of typescript issues to be fixed,
related to https://github.com/elastic/kibana/pull/166813.
It's also re-enabling full typecheck, with this, we should be back in a
clean, typechecked main branch.
Blocked by #167428
---------
Co-authored-by: Brad White <Ikuni17@users.noreply.github.com>
Co-authored-by: Brad White <brad.white@elastic.co>
Co-authored-by: Thomas Watson <watson@elastic.co>
Co-authored-by: Patryk Kopyciński <contact@patrykkopycinski.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR adds the serverless FTR tests that we already have in the [QA
quality
gate](https://github.com/elastic/kibana/blob/main/.buildkite/pipelines/quality-gates/pipeline.tests-qa.yaml#L18-L24)
to the staging quality gate.
### Details
We intentionally decided run the same set of FTR tests again in staging
for starters. We're accepting the over-testing here until we have enough
confidence and experience with our serverless product stability to
decide which set of tests to run in which environment.
This PR also explicitly sets the `EC_ENV` and `EC_REGION` environment
variables for QA and Staging. It worked fine for QA env so far without
setting the environment variable because it fell back on the QAF default
values. Setting these values explicitly, makes it more robust.
## Summary
The full typecheck would definitely fail on the on_merge job, and the
selective typecheck doesn't make much sense (if it was already through
that step in the PR.) so we're temporarily disabling this step
completely.
## Summary
After merging #167060, `Check Types` is going to fail in the on merge
pipeline until all type errors are triaged. For now, lets use the commit
diff type check.
## Summary
This PR is the core part of #166813. The original work seems to grow
large, and we'd like to enable a preventive check beforehand to prevent
more errors from entering the codebase.
The idea is to have a selective type check that would only check changed
files' projects.
- [x] when there's no extra label, run the selective type check only on
the diffing files' projects (success:
https://buildkite.com/elastic/kibana-pull-request/builds/161837)
- [x] when the label `ci:hard-typecheck` is present, run the regular
(but now, working) full typecheck (expected to fail: )
cc: @watson
---------
Co-authored-by: Brad White <brad.white@elastic.co>
Co-authored-by: Thomas Watson <w@tson.dk>
Co-authored-by: Thomas Watson <watson@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Prepares the serverless FTR tests to be runnable with a custom ES image.
(`--esServerlessImage` cli arg)
Creates a pipeline for testing and promoting ES Serverless docker
releases.
The job can be triggered here:
https://buildkite.com/elastic/kibana-elasticsearch-serverless-verify-and-promote
The three main env variables it takes:
- BUILDKITE_BRANCH: the kibana branch to test with (maybe not as
important)
- BUILDKITE_COMMIT: the kibana commit to test with
- ES_SERVERLESS_IMAGE: the elasticsearch serverless image, or tag to use
from this repo:
`docker.elastic.co/elasticsearch-ci/elasticsearch-serverless`
## TODOS:
- [x] set `latest_verified` with full img path as default
- [x] ~~find other CLIs that might need the `esServerlessImage` argument
(if the docker runner has multiple usages)~~ | I confused the `yarn es
docker` with this, because I thought we only run ES serverless in a
docker container, but `elasticsearch` can also be run in docker.
- [x] set `latest-compatible` or similar flag in a manifest in gcs for
Elastic's use-case
- [ ] ensure we can only verify "forward" (ie.: to avoid a
parameterization on old versions to set our pointers back) [on a second
thought, this might be kept as a feature to roll back (if we should ever
need that)]
There are two confusing things I couldn't sort out just yet:
#### Ambiguity in --esServerlessImage
We can either have 2 CLI args: one for an image tag, one for an image
repo/image url, or we can have one (like I have it now) and interpret
that in the code, it can be either the image url, or the tag. It's more
flexible, but it's two things in one. Is it ok this way, or is it too
confusing?
e.g.:
```
node scripts/functional_tests --esFrom serverless --esServerlessImage docker.elastic.co/elasticsearch-ci/elasticsearch-serverless:git-8fc8f941bd4d --bail --config x-pack/test_serverless/functional/test_suites/security/config.ts
# or
node scripts/functional_tests --esFrom serverless --esServerlessImage latest --bail --config x-pack/test_serverless/functional/test_suites/security/config.ts
```
#### Ambiguity in the default image path
The published ES Serverless images will sit on this image path:
`docker.elastic.co/elasticsearch-ci/elasticsearch-serverless`, however,
our one exception is the `latest-verified` which we will be tagging
under a different path, where we have write rights:
`docker.elastic.co/kibana-ci/elasticsearch-serverless:latest-verified`.
Is it okay, that by default, we're searching in the `elasticsearch-ci`
images for any tags as parameters (after all, all the new images will be
published there), however our grand default will ultimately be
`docker.elastic.co/kibana-ci/elasticsearch-serverless:latest-verified`.
## Links
Buildkite:
https://buildkite.com/elastic/kibana-elasticsearch-serverless-verify-and-promote
eg.:
https://buildkite.com/elastic/kibana-elasticsearch-serverless-verify-and-promote/builds/24
Closes: https://github.com/elastic/kibana/issues/162931
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
- Combines the `endpoint` and `mocked_data` tests suites so that they
run from the same cypress configuration/run buildkite setup
- Moved test files from the `endpoint/` and `mocked_data/` directories
into new sub-directories that more closely describe the set of tests
they contain
- The `security_solution/package.json` file was updated so that the
following `scripts` will now output a warning indicating that command is
no longer valid:
- `cypress:dw:endpoint`
- `cypress:dw:endpoint:run`
- `cypress:dw:endpoint:open`
The following npm/yarn commands remain available for running tests
locally:
```shell
yarn --cwd=x-pack/plugins/security_solution cypress:dw:open
```
```shell
yarn --cwd=x-pack/plugins/security_solution cypress:dw:run
```
New test file struncture:
<img width="415" alt="image"
src="0cb4bc76-b434-4219-b73e-508645201a81">
## Summary
- introduces tags for [Defend Workflows] cypress tests (similarly to
https://github.com/elastic/kibana/pull/162698)
- adds scripts to Security Solution:
- `cypress:dw:serverless:open` and `:run`
- `cypress:dw:endpoint:serverless:open` and `:run`
- adds CI jobs to run these scripts
- so far most of the expected tests got both `@serverless` and
`@brokenInServerless` tests, because of other issues to be solved,
- one test is able to run against serverless:
`x-pack/plugins/security_solution/public/management/cypress/e2e/mocked_data/policy_details.cy.ts`
This moves all FTR and cypress based serverless tests into the default
test pipeline. By moving these tests, failures will
now fail pull request and on-merge pipelines.
This PR reduces the parallel number of workers that will run the
Serverless Security Cypress Tests for now. As we have skipped some
serverless tests that were flaky we don't need as much workers as
before. As soon as we see those pipeline times increasing we can restore
the number of workers.
## Summary
After chatting with the team, i realized that
https://github.com/elastic/kibana/pull/165334 did not include the file
this PR is updating.
## Note
Another approach would be to keep the pipeline triggers as they are, but
make sure the e2e tests have been run in parallel with any other tests
during the CI process. This would potentially shorten the CI times since
the e2e tests take around 40 minutes to run.
**Fixes:** https://github.com/elastic/kibana/issues/165546
## Summary
This PR fixes junit report transformation for Security Solution build steps (Cypress writes reports in `mochawesome` format and `yarn junit:transform` transforms it to junit format).
## Details
After refactoring it turned out `yarn junit:merge` was moved from `cypress:*` yarn scripts to build step `*.sh` files in `.buildkite/scripts/steps/functional` folder. This way a command to run Cypress tests changed from
```sh
yarn cypress:run:ess
```
to
```sh
yarn cypress:run:ess; status=$?; yarn junit:merge && exit $status
```
In first case any test failure do not lead to early exist and all following commands are executed as yarn runs scripts without preserving shell settings. In the second case failing tests lead to `yarn cypress:run:ess` exit with non-zero exit code so the shell script exits immediately without giving a chance for `yarn junit:merge` to execute.
This problem is solved by disabling early exit on error via `set +e`.
On top of that using of `artifact_paths` config option is redundant as [post command script](https://github.com/elastic/kibana/blob/main/.buildkite/scripts/lifecycle/post_command.sh#L16) upload build artefact via a bildkite agent.
#### [Failed build example](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3049#_)
## Summary
Kibana's build jobs work on a different subset of executors than the
buildkite pipelines defined in `catalog-info.yaml`.
The former jobs require a set of `pre_command` / `post_command` steps to
prepare the jobs for building/testing kibana.
The latter don't have access rights to certain vault secrets (and
possibly other missing config from the Kibana world), but for now,
they're also not building Kibana, they just trigger other jobs, so we
can just skip the problematic hooks.
~~A probably good indicator I found for deciding whether we need the
kibana-related `pre_command` is the
`BUILDKITE_AGENT_META_DATA_AGENT_MANAGER` flag, that's set to `"kibana"`
in the case of the kibana executors.~~
We can try to match on the agent names for the CI-systems agents. They
seem to be starting with `bk-agent`.
This should allow for the
[kibana-tests](https://buildkite.com/elastic/kibana-tests) job to run.
Split from: https://github.com/elastic/kibana/pull/165346
---------
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
## Summary
The config does not propagate to downstream workflow jobs in all cases
so things break if we don't use the default. Therefore we will make dev
the exception.
Update the gpctl config we reference in the gitops repo. The default
config needs to be the promotion workflow and dev needs to be separate
otherwise other promotion infrastructure breaks since it does not know
to look for the overridden config.
https://github.com/elastic/serverless-gitops/pull/688
Slack: https://elastic.slack.com/archives/C9PPG4EJH/p1693480365606699
## Summary
Adds a pipeline that will trigger the promotion and QG for kibana
through qa -> staging -> production whenever the tag which [matches the
regex](https://regex101.com/r/tY52jo/1) is created.
Sibling PR [here](https://github.com/elastic/serverless-gitops/pull/661)
that defines `main/gen/gpctl/kibana/tagged-release.yaml`
The meat of the PR is the regex.
---------
Co-authored-by: Thomas Watson <w@tson.dk>
Co-authored-by: Alex Szabo <delanni.alex@gmail.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Closes#162593Closes#163939Closes#162625
The original intention of this PR was to add FTR support for ESS.
However the scope increased as that also required adding SSL support due
to tests failing from disabled `security` and no authentication.
Additionally, after using serverless in `kbn/es` extensively for this,
there was a bit of friction in regards to DX.
## Summary
- Switch `x-pack/test_serverless` FTR to use ES serverless instead of
(stateful) snapshot
- Adds SSL support to Docker and Serverless in `kbn/es`
- Adds `port` option override
- Adds `teardown` option to kill running nodes if the process exits
without shutdown
- Adds `kill` option to kill running nodes on startup if detected
- Adds `--esFrom serverless` to FTR CLI
- Adds `files` option to mount extra files into containers
- For serverless, automatically attach to first node with `docker logs
-f es01` on startup for better DX.
- Added `background` flag to not attach `logs`.
- Adds graceful shutdown for ESS cluster
- Separate `docker pull` from `run` for better logging, ensures latest
image and stops multiple pulls of the same image occurring in parallel
- Align (most) default settings for ES serverless with `gradlew`
[settings](https://github.com/elastic/elasticsearch-serverless/blob/main/serverless-build-tools/src/main/kotlin/elasticsearch.serverless-run.gradle.kts#L8)
- Fixes Docker bind mount permissions in CI
- Fixes issue where `esFrom` would default to `snapshot` and override
FTR config settings.
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Related Issues for Skipped Tests
Security Threat Hunting: #165135
Observability: #165138
Response Ops: #165145
---------
Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Patryk Kopycinski <contact@patrykkopycinski.com>
**Part of: https://github.com/elastic/security-team/issues/6726**
## Summary
Migrates the prebuilt rules and timelines status API route schema to
OpenAPI. This is exploratory work to assess the level of effort required
to migrate API route schemas from `io-ts` to `zod` generated by OpenAPI
codegen.
**Summary of the changes:**
- Added a CI job that runs code generation in Security Solution and
comments change if there are any.
- Migrated the `/api/detection_engine/rules/prepackaged/_status` route
to use generated `zod` schemas
- Updated schema tests
- Adjusted the code generator templates to handle `strict` schemas,
i.e., schemas that do not allow any extra params
- Updated the error transformation code to work with zod errors.
Validation errors are converted to string representations, like the
following:
<img width="627" alt="image"
src="93002573-972f-42e1-901d-01a19937f568">
## Summary
`@cypress/grep` is just reading the file as text, so it's not capable of
evaluating the variables, because of that, today we are running all the
spec files, and that means that we are setting up a new stack just to
stop it a few seconds later after grep skips the spec file
https://github.com/cypress-io/cypress/blob/develop/npm/grep/src/plugin.js#L117
With variable:
<img width="1219" alt="Zrzut ekranu 2023-08-23 o 23 06 30"
src="97b6fdaa-a03a-4a4f-bbdc-97da8e1788ce">
With string:
<img width="2074" alt="Zrzut ekranu 2023-08-23 o 23 06 04"
src="1f48d20a-3a1a-4ded-aa47-493781a4758b">
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Previously the `ci:build-serverless-image` label was hooked into the
default `Build distribution` step, which only builds the x64 linux
distribution. This limited build is in support of starting functional
tests as quickly as possible.
In order to support ARM artifacts, this starts a non-blocking parallel
build that cross compiles.
There's tradeoffs to this approach. CI time will be kept near our hour
target, at the cost of building x64 artifacts twice. Adding cross
compilation to the primary build step will extend CI time by ~30
minutes.
## Summary
Inspired by https://glebbahmutov.com/blog/burning-tests/
Implements the idea presented here
https://glebbahmutov.com/blog/burning-tests/#bonus-3-burning-new-or-changed-specs
In short, on PR that is changing/adding new Cypress spec files we will
try to "burn" them, it means we will try to run each `it` `2` times to
make sure tests are written in a way that gives Cypress a chance to
recover from the failed test.
Also adding a command that allows to "burn" tests locally
```
yarn cypress:burn --spec "<>"
```
Right now the job is set to `soft_fail`, so it is not going to block the
PR from merging, but hopefully will help the Team to recognize potential
flakiness before it is merged to `main`
Lately the build storybooks ci step is getting closer to 60 minutes
running time. For now, instead of splitting the job into multiple ones,
I think we can go with a bigger machine.
## Summary
Run Real Endpoint Cypress E2E on CI using Vagrant
---------
Co-authored-by: Tomasz Ciecierski <ciecierskitomek@gmail.com>
Co-authored-by: Ashokaditya <am.struktr@gmail.com>
