## Summary
- Combines the `endpoint` and `mocked_data` tests suites so that they
run from the same cypress configuration/run buildkite setup
- Moved test files from the `endpoint/` and `mocked_data/` directories
into new sub-directories that more closely describe the set of tests
they contain
- The `security_solution/package.json` file was updated so that the
following `scripts` will now output a warning indicating that command is
no longer valid:
- `cypress:dw:endpoint`
- `cypress:dw:endpoint:run`
- `cypress:dw:endpoint:open`
The following npm/yarn commands remain available for running tests
locally:
```shell
yarn --cwd=x-pack/plugins/security_solution cypress:dw:open
```
```shell
yarn --cwd=x-pack/plugins/security_solution cypress:dw:run
```
New test file struncture:
<img width="415" alt="image"
src="0cb4bc76-b434-4219-b73e-508645201a81">
## Summary
- introduces tags for [Defend Workflows] cypress tests (similarly to
https://github.com/elastic/kibana/pull/162698)
- adds scripts to Security Solution:
- `cypress:dw:serverless:open` and `:run`
- `cypress:dw:endpoint:serverless:open` and `:run`
- adds CI jobs to run these scripts
- so far most of the expected tests got both `@serverless` and
`@brokenInServerless` tests, because of other issues to be solved,
- one test is able to run against serverless:
`x-pack/plugins/security_solution/public/management/cypress/e2e/mocked_data/policy_details.cy.ts`
Closes https://github.com/elastic/kibana/issues/165974
The root cause of the problem is calling `basePath.prepend` on a path
without a leading `/`. Maps is not consistent with path constants. Some
have leading `/` while others do not. To resolve the issue, this PR
updates all path constants to consistently have leading `/`.
PR adds functional test runner with `map.includeElasticMapsService:
false` to ensure maps is tested without EMS enabled to catch future
regressions.
### Test instructions
* set *map.includeElasticMapsService: false* in kibana.dev.yml
* install sample data set
* create new map with documents layer
* Configure label styling
* Verify layer is displayed with labels
<img width="500" alt="Screen Shot 2023-09-13 at 6 26 23 AM"
src="44d12e87-9b80-424c-9bc9-126b373bdf18">
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This moves all FTR and cypress based serverless tests into the default
test pipeline. By moving these tests, failures will
now fail pull request and on-merge pipelines.
## Summary
This PR adds a performance journey test for the Index data visualizer,
using the Kibana TSDB logs.
To run test:
```
node scripts/run_performance.js --journey-path x-pack/performance/journeys/tsdb_logs_data_visualizer.ts --skip-warmup
```
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Walter Rafelsberger <walter.rafelsberger@elastic.co>
Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
This PR reduces the parallel number of workers that will run the
Serverless Security Cypress Tests for now. As we have skipped some
serverless tests that were flaky we don't need as much workers as
before. As soon as we see those pipeline times increasing we can restore
the number of workers.
## Summary
Fixes failed build steps when there are no tests to execute and hence no
reports.
## Details
Despite this [PR](https://github.com/elastic/kibana/pull/165824) fixes
tests failure when there are no reports but `yarn junit:merge` tries to
convert them to junit format the problem is partially stays as `Failed
Test Reporter` runs for every build step meaning also for successful
build steps. We don't need to handle failed test reports for successful
build steps. There are cases when there are no tests to run so Cypress
doesn't produce reports and nothing is converted to junit format so
`Failed Test Reporter` fails and causes a build step to fails. It could
be solved by defining an environment variable
`DISABLE_MISSING_TEST_REPORT_ERRORS=true` but we need to make sure
reports exist for failed tests. Taking this into account we can rely on
`BUILDKITE_COMMAND_EXIT_STATUS` to avoid running `Failed Test Reporter`
if the build step successed.
One may ask why don't skip `Upload Artifacts` too. We may need some
build artifacts for successful build steps.
---------
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
## Summary
Resolves: https://github.com/elastic/kibana/issues/162262
This PR is the backend changes to add a circuit breaker
`xpack.alerting.rules.maxScheduledPerMinute` to both serverless and
other environments that limits the number of rules to 400 runs / minute
and 10000 runs / minute, respectively. There will be another PR to
follow this one that gives the user UI hints when creating/editing rules
that go over this limit.
This circuit breaker check is applied to the following routes:
- Create Rule
- Update Rule
- Enable Rule
- Bulk Enable Rule
- Bulk Edit Rule
Also adds a new route: `/internal/alerting/rules/_schedule_frequency` to
get the current total schedules per minute (of enabled rules) and the
remaining interval allotment.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: lcawl <lcawley@elastic.co>
Co-authored-by: Xavier Mouligneau <xavier.mouligneau@elastic.co>
## Summary
This PR implements a temporary fix to prevent `yarn junit:merge` command
from breaking a build step by returning non zero code. `yarn
junit:merge` fails in case if it can't find reports or folders it
operates on weren't created beforehand.
## Summary
It fixes Osquery Cypress CI build step by rolling back the changes.
## Details
https://github.com/elastic/kibana/pull/165541 broke Osquery build step
by running junit transformation command while Osquery Cypress tests
don't produce any reports. The fix just removes junit transformation
command.
## Summary
After chatting with the team, i realized that
https://github.com/elastic/kibana/pull/165334 did not include the file
this PR is updating.
## Note
Another approach would be to keep the pipeline triggers as they are, but
make sure the e2e tests have been run in parallel with any other tests
during the CI process. This would potentially shorten the CI times since
the e2e tests take around 40 minutes to run.
**Fixes:** https://github.com/elastic/kibana/issues/165546
## Summary
This PR fixes junit report transformation for Security Solution build steps (Cypress writes reports in `mochawesome` format and `yarn junit:transform` transforms it to junit format).
## Details
After refactoring it turned out `yarn junit:merge` was moved from `cypress:*` yarn scripts to build step `*.sh` files in `.buildkite/scripts/steps/functional` folder. This way a command to run Cypress tests changed from
```sh
yarn cypress:run:ess
```
to
```sh
yarn cypress:run:ess; status=$?; yarn junit:merge && exit $status
```
In first case any test failure do not lead to early exist and all following commands are executed as yarn runs scripts without preserving shell settings. In the second case failing tests lead to `yarn cypress:run:ess` exit with non-zero exit code so the shell script exits immediately without giving a chance for `yarn junit:merge` to execute.
This problem is solved by disabling early exit on error via `set +e`.
On top of that using of `artifact_paths` config option is redundant as [post command script](https://github.com/elastic/kibana/blob/main/.buildkite/scripts/lifecycle/post_command.sh#L16) upload build artefact via a bildkite agent.
#### [Failed build example](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3049#_)
## Summary
Kibana's build jobs work on a different subset of executors than the
buildkite pipelines defined in `catalog-info.yaml`.
The former jobs require a set of `pre_command` / `post_command` steps to
prepare the jobs for building/testing kibana.
The latter don't have access rights to certain vault secrets (and
possibly other missing config from the Kibana world), but for now,
they're also not building Kibana, they just trigger other jobs, so we
can just skip the problematic hooks.
~~A probably good indicator I found for deciding whether we need the
kibana-related `pre_command` is the
`BUILDKITE_AGENT_META_DATA_AGENT_MANAGER` flag, that's set to `"kibana"`
in the case of the kibana executors.~~
We can try to match on the agent names for the CI-systems agents. They
seem to be starting with `bk-agent`.
This should allow for the
[kibana-tests](https://buildkite.com/elastic/kibana-tests) job to run.
Split from: https://github.com/elastic/kibana/pull/165346
---------
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
When buildkite-agent is uploading a pipeline we can skip setting up our
node environment. This stricter check avoids matching on similarly named
steps, e.g. our storybooks upload.
## Summary
This PR calls the pipeline introduced on
https://github.com/elastic/gpctl/pull/167 which calls the e2e tests
before updating serverless gitops.
The previous change which was introduced on the commits below, used
gpctl to update the image tags for kibana on the controller. This change
resulted in skipping the part where the e2e tests were being run 😬bf148fb35f3cfbf24190
The present PR fixes this, by using the new pipeline which includes the
e2e tests introduced on https://github.com/elastic/gpctl/pull/167
## Summary
The config does not propagate to downstream workflow jobs in all cases
so things break if we don't use the default. Therefore we will make dev
the exception.
Update the gpctl config we reference in the gitops repo. The default
config needs to be the promotion workflow and dev needs to be separate
otherwise other promotion infrastructure breaks since it does not know
to look for the overridden config.
https://github.com/elastic/serverless-gitops/pull/688
Slack: https://elastic.slack.com/archives/C9PPG4EJH/p1693480365606699
## Summary
Adds a pipeline that will trigger the promotion and QG for kibana
through qa -> staging -> production whenever the tag which [matches the
regex](https://regex101.com/r/tY52jo/1) is created.
Sibling PR [here](https://github.com/elastic/serverless-gitops/pull/661)
that defines `main/gen/gpctl/kibana/tagged-release.yaml`
The meat of the PR is the regex.
---------
Co-authored-by: Thomas Watson <w@tson.dk>
Co-authored-by: Alex Szabo <delanni.alex@gmail.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Closes#162593Closes#163939Closes#162625
The original intention of this PR was to add FTR support for ESS.
However the scope increased as that also required adding SSL support due
to tests failing from disabled `security` and no authentication.
Additionally, after using serverless in `kbn/es` extensively for this,
there was a bit of friction in regards to DX.
## Summary
- Switch `x-pack/test_serverless` FTR to use ES serverless instead of
(stateful) snapshot
- Adds SSL support to Docker and Serverless in `kbn/es`
- Adds `port` option override
- Adds `teardown` option to kill running nodes if the process exits
without shutdown
- Adds `kill` option to kill running nodes on startup if detected
- Adds `--esFrom serverless` to FTR CLI
- Adds `files` option to mount extra files into containers
- For serverless, automatically attach to first node with `docker logs
-f es01` on startup for better DX.
- Added `background` flag to not attach `logs`.
- Adds graceful shutdown for ESS cluster
- Separate `docker pull` from `run` for better logging, ensures latest
image and stops multiple pulls of the same image occurring in parallel
- Align (most) default settings for ES serverless with `gradlew`
[settings](https://github.com/elastic/elasticsearch-serverless/blob/main/serverless-build-tools/src/main/kotlin/elasticsearch.serverless-run.gradle.kts#L8)
- Fixes Docker bind mount permissions in CI
- Fixes issue where `esFrom` would default to `snapshot` and override
FTR config settings.
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Related Issues for Skipped Tests
Security Threat Hunting: #165135
Observability: #165138
Response Ops: #165145
---------
Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Patryk Kopycinski <contact@patrykkopycinski.com>
## Summary
The FTR serverless pipeline's output can be confusing, because in case
any of the configs fail, the error log is always attached to the last
collapsing block (thus making devs think it was the last block that was
erroneous).
This PR adds an output message after the configs are ran, and separates
the log from the last run's logs.
- chore: Add result logging for serverless_ftr.sh with a new collapsing
block to avoid confusion
Was:
<img width="1086" alt="Screenshot 2023-08-28 at 16 08 43"
src="46721879-8add-49a2-8448-b9ed40ac8bc0">
Would be:
<img width="1079" alt="Screenshot 2023-08-28 at 16 08 20"
src="3eaf7f2b-7fc8-4777-81e3-1c21ec57faaa">
**Part of: https://github.com/elastic/security-team/issues/6726**
## Summary
Migrates the prebuilt rules and timelines status API route schema to
OpenAPI. This is exploratory work to assess the level of effort required
to migrate API route schemas from `io-ts` to `zod` generated by OpenAPI
codegen.
**Summary of the changes:**
- Added a CI job that runs code generation in Security Solution and
comments change if there are any.
- Migrated the `/api/detection_engine/rules/prepackaged/_status` route
to use generated `zod` schemas
- Updated schema tests
- Adjusted the code generator templates to handle `strict` schemas,
i.e., schemas that do not allow any extra params
- Updated the error transformation code to work with zod errors.
Validation errors are converted to string representations, like the
following:
<img width="627" alt="image"
src="93002573-972f-42e1-901d-01a19937f568">
## Summary
This PR adds single user performance journey to track CRUD operations on
Tags listing page:
- get all tags on initial loading
- create a new tag
- update the tag
- delete the tag
- bulk delete (first 20 tags)
flaky-test-runner 25x:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2966
Added visualisations to monitor metrics, most operations take <1 second
<img width="523" alt="image"
src="f4c14e49-edf6-4fff-9f31-30b8a67970e9">
Since bulk delete takes ~20 sec, I put it on the separate visualisation
<img width="523" alt="Screenshot 2023-08-23 at 18 19 46"
src="467983f8-f8eb-486a-8e27-beac0d9b1f37">
dd0473ac-826f-5621-9a10-25319700326e?_g=h@3b0c329
To run locally: `node scripts/functional_tests.js --config
x-pack/performance/journeys/tags_listing_page.ts`
Note: this journey is compatible to be executed on Serverless project
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
`@cypress/grep` is just reading the file as text, so it's not capable of
evaluating the variables, because of that, today we are running all the
spec files, and that means that we are setting up a new stack just to
stop it a few seconds later after grep skips the spec file
https://github.com/cypress-io/cypress/blob/develop/npm/grep/src/plugin.js#L117
With variable:
<img width="1219" alt="Zrzut ekranu 2023-08-23 o 23 06 30"
src="97b6fdaa-a03a-4a4f-bbdc-97da8e1788ce">
With string:
<img width="2074" alt="Zrzut ekranu 2023-08-23 o 23 06 04"
src="1f48d20a-3a1a-4ded-aa47-493781a4758b">
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Currently some on merge steps fetch labels from the merged pull request
by parsing the first line of the commit message. This convention usually
works, but it overlaps with the convention of skipping test suites
linking to an issue instead of a pull request. Errors are thrown when a
step attempts to pull metadata from an unset value.
This sets a default value of an empty string if the label metadata is
unset and cleans up the log message when an error occurs.
Example error:
https://buildkite.com/elastic/kibana-on-merge-unsupported-ftrs/builds/5821#018a1d62-fbd2-486d-afe3-9a448449bbf7/257-258
## Summary
Resolves: https://github.com/elastic/kibana/issues/161788
Makes the global event log shareable. Plus some refactors like
converting the `rule_event_log_list_table` fetch to use React Query.
Also, fixed a bug with the `rule_status_panel` where we did not refresh
if the parent component refreshed.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/135309
This PR eliminates a skipped functional test by replacing the test
coverage with unit tests.
*
`x-pack/plugins/screenshotting/server/screenshots/screenshots.test.ts`:
ensures that waiting too long for the URL to open will return the
expected error message
*
`x-pack/plugins/screenshotting/server/browsers/chromium/driver.test.ts`:
ensures that when the screenshot capture method is passed an error
message, that error message is injected into the screenshot
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
It seems beneficial to have feature flag tests in a separate test config
file
- tests are still run on Kibana CI automatically
- tests are not run on MKI projects automatically, but you can deploy
custom project and run tests via feature flags config
All the feature flags within the same project should be places in the
same config to make sure there is no arguments conflict.
When the flag is moved to the yml configuration, we can rely on Kibana
CI and manually triggered deployment to make sure projects are
functioning correctly.
---------
Co-authored-by: Robert Oskamp <robert.oskamp@elastic.co>
