Lately the build storybooks ci step is getting closer to 60 minutes
running time. For now, instead of splitting the job into multiple ones,
I think we can go with a bigger machine.
## Summary
- Implement test plan as described in
`x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md`
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
We received an alert on GPCTL that a kibana commit (9509425349) did
not exist, because it was being parsed as e4 (scientific representation)
which is not what we want 😅
So, in this PR i am making sure the hash is not being parsed in any
funny way.
Relevant slack thread:
https://elastic.slack.com/archives/C03PAKL2KLK/p1689264633036939
Adding the label `ci:build-serverless-image` will build and push the
serverless image to our registry. The url will be available as an
annotation at the top of Buildkite
Closes https://github.com/elastic/kibana/issues/161562
## Summary
Run Real Endpoint Cypress E2E on CI using Vagrant
---------
Co-authored-by: Tomasz Ciecierski <ciecierskitomek@gmail.com>
Co-authored-by: Ashokaditya <am.struktr@gmail.com>
Created a separated PR in order setup a basic setup for cypress and test
https://github.com/elastic/kibana/pull/160620 for serverless.
Basic setup to run cypress for serverless-oblt
#### How to run it
from
`x-pack/test_serverless/functional/test_suites/observability/cypress`
```
yarn cypress:serverless:open
```
Reverts elastic/kibana#160900 to re-enable the pipeline that updates the
version on the single tenant service, we have checked everything, no
dying pods anymore, since we have changed the functionality to avoid
cloning kibana 😬
This PR changes how Alert Page Filter Controls Work.
## Before
1. Filter Controls use to ignore invalid Selections. For example, if
User has selected `Open` as the filter, but there is actually no alert
with Status `Open`, filters would ignore that selection and would
proceed to show other alerts ( which are NOT `Open`) .
- It seemed it was confusing users. [This
bug](https://github.com/elastic/kibana/issues/159606) and [messages in
community
slack](https://elasticstack.slack.com/archives/CNRTGB9A4/p1686937708085629?thread_ts=1686841414.978319&cid=CNRTGB9A4)
are the examples. @paulewing also emphasized this.
- Below video shows what I mean.
01771587-e4e8-4331-9535-4ffa09877c02
## After
1. With this Change Control Filters no longer ignore invalid selection.
So if user has chosen to show only `Open` Alerts. Then that filter will
be taken into account even though no alert with `Open` exists and a
empty table will be show.
- Here is a video to demonstrate it.
62b17762-16c0-471f-8480-e9f46e2ca5ef
## Summary
This PR removes standalone `common` serverless tests and instead makes
them run as part of every serverless project's tests.
### Details
Before, the `common` tests ran on a "vanilla" Kibana serverless mode
(i.e. the `serverless` plugin was loaded but none of the
`serverless-PROJECT` plugins). With continued serverless development,
this state of Kibana doesn't work as expected anymore and since this is
not supported officially anyway, it's not worth to invest making it work
properly. So we decided to accept the extra test run time and actually
include `common` tests in every project.
Reverts elastic/kibana#160508 gpctl clones the repo to verify the
commit, but since cloning kibana is not trivial, ill revert for now,
until we find a better way to handle the above on gpctl.
## Summary
This PR enables gpctl for kibana with the config definition on
serverless gitops:
https://github.com/elastic/serverless-gitops/blob/main/gen/gpctl/kibana/config.yaml
This will enable treating kibana as a service and not as a stack
component, but this will only happen on dev and only after merging:
https://github.com/elastic/serverless-gitops/pull/306
So practically for now nothing else changes. We just need the extra step
to onboard the new worfklow. Another PR will succeed this one after
verifying that everything works as expected, that will remove the
existing (legacy) step that is using a bash script instead of gpctl for
QA and Staging.
## Summary
Splitting `test/functional/apps/dashboard/group2/config.ts` as it
getting close to 35 minutes and quite often run on its own on CI worker.
<img width="1713" alt="image"
src="ac7f5dc6-2a12-4057-af98-81ff53bac1c4">`
This PR splits config into 2 almost run time equal groups:
- test/functional/apps/dashboard/group2/config.ts 18m 31s
- test/functional/apps/dashboard/group6/config.ts 16m 53s
Flaky-test-runner for both configs:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2522
## 📓 Summary
Closes https://github.com/elastic/observability-dev/issues/2655
This PR introduces a customized log consumption experience in the
Discover plugin. By leveraging the new `discover_log_explorer` plugin
and utilizing the `discover.customize` functionality, we have curated a
more tailored user experience.
The key feature of this implementation is the `DatasetSelector`
component, which replaces the original Discover `DataViewPicker`. It
handles the retrieval, rendering, and navigation of integrations and
data streams related to logs, providing an improved user interface.
This PR involves significant development efforts, including the creation
of the `discover_log_explorer` plugin, implementation of services, state
machines, custom hooks, and enhancements to presentational components.
The following overview will help reviewers understand the
responsibilities of each component in this implementation.
d725b699-452e-4718-8189-8dc1fab4d044
## DatasetsService & DatasetsClient
The DatasetsService is introduced, a crucial component that mediates
access to the newly implemented DatasetsClient. During the plugin's
lifecycle, the DatasetsService exposes a client property through its
start() method, providing convenient access to a DatasetsClient
instance.
The DatasetsClient is responsible for abstracting the data fetching
process for two endpoints: the integrations endpoint and the data
streams listing endpoint. These endpoints are utilized to populate the
selector options in the user interface. To facilitate this, the
DatasetsClient exposes the findIntegrations and findDatasets methods,
which handle the respective data fetching.
## Discover Customization
The critical part of this work consists of where the customization is
applied.
Inside the `public/plugin.tsx`, we lazy load and create, injecting the
required dependencies, the `CustomDatasetSelector`, which already
encapsulates all the logic required to make the selector work with the
external APIs.
We kept separating the data fetching logic from how the selector works,
and all the data and events are passed into the UI component with
properties.
```ts
discover.customize(
DISCOVER_LOG_EXPLORER_PROFILE_ID,
({ customizations, stateContainer }) => {
customizations.set({
id: 'search_bar',
CustomDataViewPicker: createLazyCustomDatasetSelector({
datasetsClient: datasetsService.client,
stateContainer,
}),
});
...
```
## Data fetching state machines & custom hooks
To handle the data fetching of integrations and unmanaged data streams,
we created two different state machines to separately handle the related
action for each dataset, such as remote search, in-memory search, error
handling etc.
### Integration machine and useIntegrations
The integrations state machine handles automatic data fetching of the
resources and additionally provides transitions for loading more
integrations, searching integrations by HTTP request, searching locally
into integration streams, and all the related loading and error handling
states.
It is then interpreted inside the `useIntegrations` custom hook, which
exposes the fetched data and handlers for all the above-mentioned
actions.
<img width="1975" alt="Screenshot 2023-05-30 at 09 44 42"
src="6daeca9f-826d-4a0f-bd90-eb4826ed1bde">
### Datasets machine and useDatasets
Similar to the integrations state machine, but simplified since the data
streams search can only happen with HTTP requests and there is no
pagination that requires to handle the load of more entries.
It is interpreted inside the `useDatasets` custom hook, which also
exposes the fetched data and handlers for the available actions.
<img width="1692" alt="Screenshot 2023-05-30 at 09 45 11"
src="5f9690e2-4e8f-439e-9ffd-f3b34cf3eaf5">
## DatasetSelector
The `DatasetSelector` component contains all the logic that manages the
navigation and searches across the different panels that render
integrations, integrations' streams or unmanaged streams.
As the datasets come from different APIs or are performed in-memory, the
search work follow this logic:
- When listing the integrations list (first level of the
`EuiContextMenu`), the search is done with an HTTP request.
- When listing the data streams list for a specific integration (second
level of the `EuiContextMenu`), the search is done in-memory, filtering
and sorting directly in the client.
- When listing the unmanaged data streams list (second level of the
`EuiContextMenu`), the search is done again with an HTTP request.
To handle these possible user journeys correctly without side effects,
we created another state machine and exposed its actions with an
internal `useDatasetSelector` custom hook.
<img width="1978" alt="Screenshot 2023-05-30 at 09 46 04"
src="84aa4247-c65d-40de-9eb6-6117bee731f8">
## Next steps
This component will change quite a lot until we won't get to a final
design. As soon as a first solid mvp is defined for production, a
complete test for the component will be implemented, among with a more
generic functional test for the core customization features.
---------
Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
Co-authored-by: Felix Stürmer <weltenwort@users.noreply.github.com>
## Summary
Fix https://github.com/elastic/kibana/issues/160385
Use the internal client instead of the scoped one for the extended stats
ES requests to avoid an error with unauthenticated users (when anonymous
access is allowed)
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Relates to https://github.com/elastic/kibana/issues/159451.
This PR adds api integration tests to:
- [x] GET /internal/observability_onboarding/custom_logs/privileges
- [x] GET
/internal/observability_onboarding/custom_logs/install_shipper_setup
- [x] POST /internal/observability_onboarding/custom_logs/save
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
As part of the actions for making Profiling production ready, this PR
adds basic API tests on the Profiling APIs checking if only users with
`access:profiling` are allowed to call our APIs, other users must be
forbidden.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Context: https://elastic.slack.com/archives/C0D8P2XK5/p1687424169737829
The flaky test runner was displaying incorrect name on the tasks on
Buildkite.
The reason was a partial match (cypress/security_solution_investigations
"includes" `security_solution`), now changed to a full match on the
keys.
## Summary
Update Osquery tests to changes done in
https://github.com/elastic/kibana/pull/159733
Add Osquery Cypress to `on_merge_unsupported_ftrs.yml` to get
notifications once tests are failing
---------
Co-authored-by: Tomasz Ciecierski <ciecierskitomek@gmail.com>
Co-authored-by: Tomasz Ciecierski <tomasz.ciecierski@elastic.co>
## Summary
Fix https://github.com/elastic/kibana/issues/158910
Changes the behavior of the `/api/status` endpoint to always returns a
consistent http status code, and in particular:
- during the preboot stage
- when accessed by unauthenticated users and `status.allowAnonymous` is
`false`.
That way, `/api/status` can properly be used for readiness checks.
Please refer to https://github.com/elastic/kibana/issues/158910 for more
details.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
[This PR copies off of this one which adds a config for Investigations,
but instead adds one for
Explore.](https://github.com/elastic/kibana/pull/158236)
In this PR following this issue:
https://github.com/elastic/kibana/issues/153661 we are creating a
Cypress execution just for the `Explore` team. That would help the teams
to improve their ownership of the tests.
- This PR moves one Explore test to the new directory. We plan to move
them all in a follow up PR.
- We are updating the codeowners file to reflect this new change.
- We are creating a new script that runs just the explore team.
- We are doing all the necessary changes to have all the tests inside
the new explore folder executed on the CI
- We are adding this new execution to the unsupported ftrs execution
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
After a pull request has been merged, the `kibana-on-merge` pipeline
will
trigger a new pipeline, `kibana-serverless`. This pipeline contains our
serverless tests and will initially be non-blocking.
These tests will be merged into `kibana-on-merge` when proven stable.
https://buildkite.com/elastic/kibana-serverless
Adds four pipeline steps for running serverless common, observability,
search, and security suites on pull requests.
While tests go through a stabilization period, configs will need to be
added `serverless_ftr.sh` in addition to `ftr_configs.yml`. These tests
will be allowed to fail without causing a build failure. After
stabilization and integration with the primary test pipeline, we can
revert these changes.
# Prevent Security Solution Cypress tests from parallelizing when run by
the flaky test runner
In pull request and other pipelines, the Cypress security solution
runner is expected to support parallel jobs runs. Several jobs are
spawned with the same command, but with different environment variables.
Each jobs runs a subset of the tests based on the environment variables.
These variables are BUILDKITE_PARALLEL_JOB_COUNT and
BUILDKITE_PARALLEL_JOB.
The Flaky Test Runner is an app that runs many instances of the same
config concurrently in order to determine if any of the tests in the
config are flaky. When the Flaky Test Runner runs our cypress tests,
each job should run all tests instead of a subset.
The flaky test runner sets the following environment variables to
prevent jobs from running a subset of tests:
```
// by setting chunks vars to value 1, which means all test will run in one job
CLI_NUMBER: 1,
CLI_COUNT: 1,
```
These environment variables aren't respected by the Security Solution
Cypress tests. This is also true for other Cypress tests that use the
same code:
* cypress/security_solution_investigations
* cypress/osquery_cypress
This PR introduces a new environment variable: RUN_ALL_TESTS, which
explicitly tells the runner to run all tests instead of a subset. This
causes the runner to ignore BUILDKITE_PARALLEL_JOB_COUNT and
BUILDKITE_PARALLEL_JOB
### Note about other cypress tests
This environment variable will be passed to other Cypress runners that
aren't used or maintained by Security. Specifically:
* cypress/fleet_cypress
* cypress/apm_cypress
Fleet does not appear to support parallelism, while APM does. In the APM
script, a IS_FLAKY_TEST_RUNNER environment variable is created by
reading CLI_COUNT:
09902b1055/.buildkite/scripts/steps/functional/apm_cypress.sh (L13)
```
IS_FLAKY_TEST_RUNNER=${CLI_COUNT:-0}
```
The CLI_COUNT variable isnt affected by this PR, so this should still
work, but it would be possible to use the same variable for both
scripts.
### Other Ways we could do this
We could also modify the our cypress runner to respect CLI_COUNT and
CLI_NUMBER instead of the buildkite parameters. We cannot override the
buildkite parameters via the step definition as that is not supported.
### Validation
When running our Cypress tests via the test runner, it may not be
immediately apparent that there is an issue.
This PR is equivalent to main except a change to a README:
https://github.com/elastic/kibana/pull/159301
Here is a flaky test runner build using that PR. It runs 25 jobs of the
Investigation cypress e2e tests:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2368
Each job should run the same tests (so we can check if they are flaky.)
However job 1 runs these:
* alert_table_action_column.cy.ts
* alerts_cell_actions.cy.ts
<img width="747" alt="image"
src="2090cf9b-0a88-4b28-b7fa-13ce77aa26a3">
And job 2 runs these:
* building_block_alerts.cy.ts
<img width="733" alt="image"
src="bc2f672f-ac48-4f8f-8b8b-0997c2aff874">
#### With the fix
Here is a flaky test runner build, also 25 jobs of the Investigation
cypress e2e tests, but using this PR:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2369
We have some failing tests, as our suite is currently flaky, however you
can see that each job runs all of the tests.
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Split our current FTR config in two so each one runs in a separated
config.
Moved all endpoint config related FTR tests to a new FTR config.
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
The Flaky Test Runner is a feature of the CI stats server that creates a
buildkite job to run a test suite lots of times. This can help debug
flaky tests. Currently Investigations Cypress e2e tests cannot be run in
the flaky test runner. With this PR, the Investigations e2e tests can be
run in the flaky build runner.
See https://ci-stats.kibana.dev/trigger_flaky_test_runner
### Developer validation
In order to validate this, you'll need to create your own PR with all
the changes in this PR. You can do that like so:
```
gh pr checkout https://github.com/elastic/kibana/pull/159247
git checkout -b my-branch
git push -u fork # where fork is the git remote for your own fork
```
Then follow the link to create a PR.
Next, navigate to https://ci-stats.kibana.dev/trigger_flaky_test_runner
and select your PR. Then in the second step, you should be able to type
'investigations' and see the Investigations Cypress tests.
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
This pull request introduces parallel functionality to the TI Cypress
tests, which was recently implemented in the Security Solution. In
essence, this allows us to run multiple test files simultaneously, each
within its own Kibana/ES environment. The PR is here
https://github.com/elastic/kibana/pull/157387
The changes in the scope of the TI Cypress e2es include:
- Modified `package.json` scripts; `:run` is now being used both locally
and in CI. Note that Firefox is currently not supported.
- Removed all `{testIsolation: false}` flags, as no test should depend
on non-pure state.
- All actions required to set up the test environment (archiver, login)
have been moved to beforeEach blocks to ensure that each test starts
with a fresh state.
- Introduced ESLint configuration to help us keep up with the best
practices when writing tests.
TODO:
- The `timeline.cy.ts` test is flaky and requires refactoring. There are
too many actions being fired against elements that are not visible to
the user (`{force: true}`). I've added some waits to address race
conditions in the UI, but this should be considered a temporary
solution.
---------
Co-authored-by: Patryk Kopyciński <contact@patrykkopycinski.com>
Co-authored-by: PhilippeOberti <philippe.oberti@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
In #155284 we created a new image type containing several serverless
specific configuration files. The previous distribution name was
preserved to keep the changeset limited. This is the followup for
renaming the image prefix from kibana to kibana-serverless.
This PR addresses Cypress parallelisation in Security Solution scope. It
is a first step in improving e2e tests reliability and gives us a solid
foundation for further work on flakiness and run times. With this PR
each CI job starts 3 instances of ES, kibana and Cypress runner in
isolation.
Other issues addressed in this PR:
- Use click() instead of trigger('click') in all Cypress tests.
- Use testIsolation in all Cypress tests.
- login before each test to ensure that the correct user is always
logged in. Also, with test isolation enabled, login should be required
before each test.
- use visit() instead of cy.visit() in some cases so that tests wait for
the page to fully load before executing.
- Cypress e2e tests are no longer compatible with Firefox (or any
non-Chromium based browser) due to using cypress-real-events
- Removes uses of cypress-pipe. Tests that used cypress-pipe in order to
retry click events will no longer retry click events.
- Fixes an error related to test files being overwritten by Webpack file
processing
---------
Co-authored-by: Patryk Kopycinski <contact@patrykkopycinski.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
