n2-2-spot preemption are frequent enough that this is causing pipeline
instability. This moves the instance size up to n2-4
This also removes the max-old-space-size definition I previously set.
delanni helped me understand that we're spawning a tsc process with a
memory limit already defined.
Currently on CI, the precommit_hook will fix lint warnings against the
most recent commit. When multiple commits are pushed up at once, lint
warnings on earlier commits may go unnoticed until they are squashed and
merged. The on-merge pipeline will fail due to changed files, and pull
requests based on the squashed commit will see unrelated changes
auto-fixed.
This removes the --fix flag from the precommit hook check.
Closes https://github.com/elastic/kibana/issues/169966
## Summary
Continuation on: #167980
Now that we have the snapshots created for merges, we can compare the
existing snapshots.
This PR creates a CLI for grabbing and comparing these snapshots.
The CLI looks like this:
```
node scripts/snapshot_plugin_types compare --from <rev|filename|url> --to <rev|filename|url> [--outputPath <outputPath>]
Compares two Saved Object snapshot files based on hashes, filenames or urls.
Options:
--from The source snapshot to compare from. Can be a revision, filename or url.
--to The target snapshot to compare to. Can be a revision, filename or url.
--outputPath The path to write the comparison report to. If omitted, raw JSON will be output to stdout.
--verbose, -v Log verbosely
```
Fix https://github.com/elastic/kibana/issues/147935
Automatically run the SO def integration test, and commit the changes if
any, as a new CI check script.
As explained in the issue, this is just some DX improvement, avoiding
for the developer the inconvenience of doing it manually every time they
change or add a SO in any PR.
Currently operations does not have a good mechanism for communicating
soft failures, leading to confusion on pull requests. We're tracking
support for soft failures at
https://github.com/elastic/kibana/issues/169244.
For example, in https://github.com/elastic/kibana/pull/169309, it
doesn't makes sense for the author to be responsible for fixing
flakiness here, so we need to make it clear it can be ignored or find a
way to skip this step in similar situations.
This moves the cypress burn out of the base pull request pipeline,
behind the `ci:all-cypress-suites` and `ci:cypress-burn` labels. I'm
open to all improvements here, not sure this quite finds the right
balance between stability and noise.
Introduced in https://github.com/elastic/kibana/pull/157334
This pipeline was/(is?) used for testing the new buildkite system. We're
running into an issue where `trigger_comment_regex` matches on both the
legacy and new systems, triggering builds in both. GitHub checks can end
up conflicting making it difficult to find the correct build link.
This disables the pr integration for kibana-kme-test. If we're done with
the testing pipeline we can remove it, if not we can change the trigger
word. I'm not sure what the current status is, ping @elastic/ci-systems
.
- Adds an entrypoint, `node scripts/yarn_deduplicate` for fixing
duplicated packages via npm package `yarn-deduplicate`
- Fixes existing duplicated packages
- Adds a CI check verifying no duplicated packages have been added.
There are currently two exclusions - all types packages which causes
other checks to break, and axe-core, which does not follow semver.
Closes https://github.com/elastic/kibana/issues/125712
These tests are not sharded across agents, and historical data is not
captured for flaky tests. I'm leaving the pull request parallelism flags
as is in case the goal is to prevent flakiness, but in both cases
operations strongly prefers integration with the flaky test runner
instead.
After creating a new serverless image off the main branch, in addition
to pushing an image tagged with `git-<sha>` this also pushes
`docker.elastic.co/kibana-ci/kibana-serverless:latest`.
## Summary
- Following the initial work in this
[PR](https://github.com/elastic/kibana/pull/166755)
- Addresses part of https://github.com/elastic/kibana/issues/151902 for
exceptions
-
https://docs.google.com/document/d/1CRFfDWMzw3ob03euWIvT4-IoiLXjoiPWI8mTBqP4Zks/edit
first two rows in the new groups
- Relocated the tests promptly without focusing on refactoring, as our
priority was to obtain coverage as well as grouping them
- Transferred the relevant utilities and updated 'signal' to 'alert.'
- Temporarily marked security role-related tests as 'brokenInServerless'
until resolution.
- Moved tests linked to exceptions requiring a `trail` in ESS and
`complete` in Serverless under a new folder called `default` license.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Security runs a code generator script on PRs. It only takes a few
seconds. This change moves it to the 'checks' command. This will save on
build agents.
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
Co-authored-by: Jon <jon@elastic.co>
Security solution's serverless QA quality gates which were added in
https://github.com/elastic/kibana/pull/167494 are failing when trying to
run cypress tests. I've marked them to soft fail here so we can unblock
the serverless release pipeline.
cc @watson @elastic/kibana-operations @MadameSheema @charlie-pichette
@YulNaumenko
## Summary
#### Code update:
`/detection_engine/routes/signals/set_alert_tags_route.ts`
- When enabling the alert tags cypress test, noticed it was failing as
the tag updates did not show in the UI until a few seconds later upon a
second refresh. I was able to recreate this locally on serverless, not
on ESS. I updated the alerts tag route to include `refresh: true` and
that seemed to fix this issue.
- `/detection_engine/routes/signals/open_close_signals_route.ts`
- When testing on serverless, alert status was stale after update.
Confirmed this with tests that were failing for ESS. Upon updating route
to use `refresh: true`, tests began passing and could see expected
behavior. This may make the call a bit heavier so we will want to see if
there are any performance impacts.
## Summary
Currently, we have our cypress tests properly integrated with the first
quality gate.
In this quality gate, our tests are executed as part of the PR process
using the serverless FTR environment. This environment uses the latest
elasticsearch snapshot and the kibana source code, but it is not a real
serverless environment because is not a deployed project on the cloud.
In order to continue assuring that we don’t introduce new issues, we
want to execute automated tests in the second quality gate as well. The
second quality gate is a deployed serverless project in the QA
environment.
We want to start moving slowly in that direction and at the same time
make sure we don't introduce flakiness, and the tests perform well.
In this PR we are creating the `@serverlessQA` tag. Tests including that
label will be executed on the second quality gate. We are also adding
the label to one of the tests we know that is stable in serverless to
test the full integration.
Note that currently, we have some known limitations in this gate pending
to be solved and we are working on it:
- Execution of the tests is sequential
- No differentiation between teams during the execution is done over the
AET umbrella
- The environment is set in `complete` so tests that exercise the
`essential` behaviour should not be added.
- Please ping me in case you want to add more tests into that gate to
asses the risk before we have the gate tested and all the serverless
tests stabilized.
---------
Co-authored-by: Georgii Gorbachev <banderror@gmail.com>
Co-authored-by: dkirchan <diamantis.kirchantzoglou@elastic.co>
# Summary
- Addresses https://github.com/elastic/kibana/issues/161537
## Description
- This PR follows the second option defined in this
[document](https://docs.google.com/document/d/1mqkpjDdFQRFvx_RPvNmjstVj8SXYMr2mrETMv3esda8/edit#heading=h.rpv1zyeb04ay)
the [Mocha tagging ](https://github.com/mochajs/mocha/wiki/Tagging)
- It introduces a new folder
`x-pack/test/security_solution_api_integration` which will serve as a
centralized location to meet all the requirements related to renaming
tests appropriately and grouping similar tests together. It will
facilitate the management of tests that must be run in Serverless and
ESS environments.
- Within this folder, there is a "config" subdirectory that stores base
configurations specific to both the
[Serverless](https://github.com/elastic/kibana/pull/166755/files#diff-afe1f42d5ac2006de8dc09069448b9e8734a6a950586376cd6e8eeb9110ab5f1R1)
and
[ESS](https://github.com/elastic/kibana/pull/166755/files#diff-4a60bd8c91da08a3f7ec14bf3bfef8449af155611374c32579b0318da03e292cR1)
environments, These configurations build upon the base configuration
provided by test_serverless and api_integrations, incorporating
additional settings such as environment variables and tagging options.
- It demonstrates scenarios involving `@ess`, `@serverless`, and
`@brokenInServerless`.
- The file`
x-pack/test/security_solution_api_integration/test_suites/detections_response/rule_creation/create_rules.ts`
is functional in both **Serverless** and **ESS**. However, some tests
related to roles are currently skipped for Serverless until they are
resolved, and these tests are tagged with `@brokenInServerless`
## CI
- It includes a new entry in the ftr_configs.yml to execute the newly
added tests in the pipeline.
- It involves the addition of mochaOptions in both
serverless/config.base.ts and ess/config.base.ts. In the case of
serverless, it includes **@serverless** while excluding
@brokenInServerless. Similarly, for **ess**, it includes @ess and
excludes **@brokenInEss**.
from `x-pack/test/security_solution_api_integration/config/serverless`
## Update in x-pack/test/detection_engine_api_integration
- The `create_rules.ts` and `create_rule_exceptions` files have been
relocated from
`x-pack/test/detection_engine_api_integration/security_and_spaces/group1`
to their respective domains within the
`x-pack/test/security_solution_api_integration` folder.
- The util files now are copied over from the old folder
`x-pack/test/detection_engine_api_integration` to the new folder and
will be removed once all tests are moved to the new folder to don't
break the existing tests
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Allows for env-var controlled filtering of executed test suites in the
ES Serverless verification job, and adjusts existing flags and behaviour
to better-fit future usage.
We're changing the job to not publish the `latest-verified` tag by
default. This is to prevent external calls to the job from accidentally
promoting random tags to `latest-verified`, see changes below.
Flag changes:
- `PUBLISH_DOCKER_TAG`: if set to 1/true, passing runs will promote the
tested ES Serverless tag to `latest-verified`.
- (used to be default, now it requires this flag)
- `PUBLISH_MANIFEST`: if set to 1/true, passing runs will upload the
manifest attesting what (kibana + es) combination was used in the test
- (used to be called `UPLOAD_MANIFEST`)
- `SKIP_CYPRESS`: if set to 1/true, it will skip running the cypress
tests
- new flag
- `FTR_EXTRA_ARGS`: a string argument, if passed, it will be forwarded
verbatim to the FTR run script
- new flag, can be used to control the filtering required for #167611
(eg.: `FTR_EXTRA_ARGS="--include-tag=ml"`)
Example run:
https://buildkite.com/elastic/kibana-elasticsearch-serverless-verify-and-promote/builds/64#018b1a30-3360-4995-874a-864f18e104d5Closes: #168376
## Goal
We'd like to increase visibility to when Saved Object migrations and
schema changes are added to serverless releases.
## Plan
- add post-build step to export Saved Object schema snapshot, upload it
to google storage, by commit hash.
- build comparison logic, that when given 2 hashes (e.g.: #current ->
#to-be-released) it can highlight schema changes
- display the output from the comparison where it would make most sense,
ideas:
- prior to release, as a script that can be ran (or a browser-based
inspector)
- prior to release, as a buildkite step, that displays the output, and
waits for a confirmation on it
- during the quality-gating, as an output during the quality-gate run,
to be confirmed by the RM
## Summary
The PR intends to satisfy the first step of the plan, to add a
post-merge step to snapshot SO schema/migration states for later use.
Example run:
https://buildkite.com/elastic/kibana-pull-request/builds/165992#018b0583-c575-47f4-bade-7b45f6cf3f4d
The files are not public, they require google cloud access, but could be
accessed by scripts we create.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This pull request tackles flaky `artifacts.cy.ts` e2e test. Ran 7+ times
without flakiness in CI.
Added Test Burning to Defend Workflows pipeline.
Changed:
1. Removed `.first()` selectors in favour of explicit element getters
2. Removed method chain
3. Moved url change from `before` to test body.
4. In `beforeAll` now we first fetch current revision number, then run
`Promise.all` on `removeAllArtifacts` which resolves with `true/false`
based on whether they actually removed an artifact. Its being counted
then and with this number we know what next revision should be (revision
fetched before removal + times removal was successful). We query API
recursively until revisions match and only then we can be sure that
displayed in the UI revision is up to date. This fixes the main reason
of the flakiness of this test and makes it a pure test that should
always yield the same result no matter the number of runs.
---------
Co-authored-by: Patryk Kopyciński <contact@patrykkopycinski.com>
## Summary
After [164798](https://github.com/elastic/kibana/pull/164798) was merged
CI singnificantly slowed down because f these FTR groups:
```
x-pack/test_serverless/functional/test_suites/security/common_configs/config.group2.ts: 75.6 minutes
x-pack/test_serverless/functional/test_suites/search/common_configs/config.group2.ts: 75 minutes
x-pack/test_serverless/functional/test_suites/observability/common_configs/config.group2.ts: 74.7 minutes
```
This PR splits each group into 3.
## Summary
Security Solution writes e2e and other tests using Cypress. In the past,
these tests, if they failed on a tracked branch, couldn't be easily
skipped. They also weren't run in parallelized jobs. For primarily these
reasons, they didn't run on most Kibana PRs.
This PR moves these Cypress tests back to the main PR pipeline. Tests
that fail on tracked branches create (or update) Github issues which can
be used with the skip-test github workflow script to easily skip the
failing tests. The pipeline steps are parallelized and run in under 40
minutes.
### Open Questions
- [ ] Should this PR enable Serverless Security Defend Workflows Cypress
Tests @patrykkopycinski
### Some buildkite pipelines that used to run only on Security PRs now
run on all PRs:
These steps run on all PRs with these changes
- Security Solution Cypress Tests (general tests that haven't been
organized into an area team)
- Explore tests
- Investigations Tests
- Defend Workflows Tests
- Defend Workflows Serverless
- Threat Intel Tests
- OS Query Tests
- Security Solution Burning Changed Specs (these run only recently
changed specs a few extra times)
- Security Solution OpenAPI codegen
- OSQuery burning
- OSQuery Serverless
<details>
<summary><b>And these already run on all PRs</b></summary>
- Serverless Security Cypress Tests
- Serverless Explore tests
- Serverless Investigations Tests
</details>
### Security Cypress tests run in the main `on merge` pipeline instead
of the `on merge unsupported ftrs` pipeline:
These steps run in the `on merge` pipeline with these changes:
- Security Solution Cypress Tests
- Explore Cypress Tests
- Investigations Cypress Tests
- Defend Workflows Cypress Tests
- Defend Workflows Serverless Cypress Tests
- Threat Intelligence Cypress Tests
- Osquery Cypress Tests
<details>
<summary><b>and these already run on the `on merge`
pipeline</b></summary>
- Serverless Security Cypress Tests
- Serverless Explore - Security Solution Cypress Tests
- Serverless Investigations - Security Solution Cypress Tests
</details>
### Additional work to be done:
We need to consolidate build steps, enhance test skipping to support
Cypress-grep flags, avoid out-of-memory errors in cypress, enhance
parallelization, improve Cypress reporting, and probably other things.
These are tracked separately. Reach out to me if you need details.
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
