Commit graph

1469 commits

Author SHA1 Message Date
Alex Szabo
2fbc843e0d
[ci] Extract OAS check + add retry (#196534)
## Summary
In the past months, the capture OAS snapshot step has been quite flaky,
breaking the `Checks` step, and thus breaking the `on-merge` jobs.

This PR extracts the check to its own step and adds retries. The
separate step is ideal because it is quite heavy compared to the other
checks (we fire up ES + Kibana for the OAS snapshot).

Also, this PR removes the `Checks` step altogether from the
kibana-chrome-forward-testing pipeline, as the Chrome versions used do
not affect that aspect.

This test run includes a retry within the Capture OAS Snapshot step:
https://buildkite.com/elastic/kibana-pull-request/builds/243187#01929612-dac7-4584-b440-120ea3fea7ea
2024-10-17 15:03:28 +02:00
Nathan L Smith
9131fbe2d2
Add obs inventory storybook to aliases (#195843)
This makes it so observability inventory storybooks work with `yarn
storybook` and are published to
https://ci-artifacts.kibana.dev/storybooks/main/latest/index.html and PR
builds.
2024-10-16 16:22:46 -05:00
Jatin Kathuria
2eff6d1046
[ Security Solution ]Re-organize the Investigations api integration test for MKI (#194707)
## Summary

This is Part 1 of the in resolving the issue :
https://github.com/elastic/kibana/issues/183645 .

This PR re-organizes investigations API tests so that they can be run in
Serveless MKI at both `basic/essentials` and `complete` licenses.

## How to test this PR 

Below are the commands that are affected by this change and you can test
the PR by running below commands.

Each commands sets up the test environment and give you a command to run
tests. Please run those tests to see if everything is okay. An example
is shown in below screenshot.

<img width="1916" alt="grafik"
src="https://github.com/user-attachments/assets/fa400450-e4aa-41dc-a1ea-ac21634c46d3">


|Module|Deployment|License|Command|
|--|--|--|--|
|Timelines|ESS|basic|`yarn investigations:basic:timeline:server:ess`|
|Timelines|ESS|Trial|`yarn investigations:timeline:server:ess`|
|Timelines|Serverless|basic|`yarn
investigations:basic:timeline:server:serverless`|
|Timelines|Serverless|Trial|`yarn
investigations:timeline:server:serverless`|
|Saved Objects|ESS|basic|`yarn
investigations:basic:saved-objects:server:ess`|
|Saved Objects|ESS|Trial|`yarn investigations:saved-objects:server:ess`|
|Saved Objects|Serverless|basic|`yarn
investigations:basic:saved-objects:server:serverless`|
|Saved Objects|Serverless|Trial|`yarn
investigations:saved-objects:server:serverless`|

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-10-16 12:56:41 -05:00
Jon
8afbbc0082
[ci] Add kibana-vm-images pipeline (#195816) 2024-10-15 07:40:46 -05:00
Aleh Zasypkin
cb2112cae5
feat: allow plugins to deprecate and replace features and feature privileges (#186800)
## Summary

This change is the implementation of the `Kibana Privilege Migrations`
proposal/RFC and provides a framework that allows developers to replace
an existing feature with a new one that has the desired configuration
while teaching the platform how the privileges of the deprecated feature
can be represented by non-deprecated ones. This approach avoids
introducing breaking changes for users who still rely on the deprecated
privileges in their existing roles and any automation.

Among the use cases the framework is supposed to handle, the most common
are the following:

* Changing a feature ID from `Alpha` to `Beta`
* Splitting a feature `Alpha` into two features, `Beta` and `Gamma`
* Moving a capability between privileges within a feature (top-level or
sub-feature)
* Consolidating capabilities across independent features

## Scope

This PR includes only the core functionality proposed in the RFC and
most of the necessary guardrails (tests, early validations, etc.) to
help engineers start planning and implementing their migrations as soon
as possible. The following functionality will be added in follow-ups or
once we collect enough feedback:

* Telemetry
* Developer documentation
* UI enhancements (highlighting roles with deprecated privileges and
manual migration actions)

## Framework

The steps below use a scenario where a feature `Alpha` should be split
into two other features `Beta` and `Gamma` as an example.

### Step 1: Create new features with the desired privileges

First of all, define new feature or features with the desired
configuration as you'd do before. There are no constraints here.

<details>

<summary>Click to see the code</summary>

```ts
deps.features.registerKibanaFeature({
  id: 'feature_beta',
  name: 'Feature Beta',
  privileges: {
    all: {
      savedObject: { all: ['saved_object_1'], read: [] },
      ui: ['ui_all'],
      api: ['api_all'],
      … omitted for brevity …
    },
    read: {
      savedObject: { all: [], read: ['saved_object_1'] },
      ui: ['ui_read'],
      api: ['api_read'],
      … omitted for brevity …
    },
  },
  … omitted for brevity …
});

deps.features.registerKibanaFeature({
  id: 'feature_gamma',
  name: 'Feature Gamma',
  privileges: {
    all: {
      savedObject: { all: ['saved_object_2'], read: [] },
      ui: ['ui_all'],
      // Note that Feature Gamma, unlike Features Alpha and Beta doesn't provide any API access tags
      … omitted for brevity …
    },
    read: {
      savedObject: { all: [], read: ['saved_object_2'] },
      ui: ['ui_read'],
      // Note that Feature Gamma, unlike Features Alpha and Beta doesn't provide any API access tags
      … omitted for brevity …
    },
  },
  … omitted for brevity …
});
```

</details>

### Step 2: Mark existing feature as deprecated

Once a feature is marked as deprecated, it should essentially be treated
as frozen for backward compatibility reasons. Deprecated features will
no longer be available through the Kibana role management UI and will be
replaced with non-deprecated privileges.

Deprecated privileges will still be accepted if the role is created or
updated via the Kibana role management APIs to avoid disrupting existing
user automation.

To avoid breaking existing roles that reference privileges provided by
the deprecated features, Kibana will continue registering these
privileges as Elasticsearch application privileges.

<details>

<summary>Click to see the code</summary>

```ts
deps.features.registerKibanaFeature({
  // This is a new `KibanaFeature` property available during feature registration.
  deprecated: {
    // User-facing justification for privilege deprecation that we can display
    // to the user when we ask them to perform role migration.
    notice: i18n.translate('xpack.security...', {
      defaultMessage: "Feature Alpha is deprecated, refer to {link}...",
      values: { link: docLinks.links.security.deprecatedFeatureAlpha },
    })
  },
  // Feature id should stay unchanged, and it's not possible to reuse it.
  id: 'feature_alpha',
  name: 'Feature Alpha (DEPRECATED)',
  privileges: {
    all: {
      savedObject: { all: ['saved_object_1', 'saved_object_2'], read: [] },
      ui: ['ui_all'],
      api: ['api_all'],
      … omitted for brevity …
    },
    read: {
      savedObject: { all: [], read: ['saved_object_1', 'saved_object_2'] },
      ui: ['ui_read'],
      api: ['api_read'],
      … omitted for brevity …
    },
  },
  … omitted for brevity …
});
```
</details>

### Step 3: Map deprecated feature’s privileges to the privileges of the
non-deprecated features

The important requirement for a successful migration from a deprecated
feature to a new feature or features is that it should be possible to
express **any combination** of the deprecated feature and sub-feature
privileges with the feature or sub-feature privileges of non-deprecated
features. This way, while editing a role with deprecated feature
privileges in the UI, the admin will be interacting with new privileges
as if they were creating a new role from scratch, maintaining
consistency.

The relationship between the privileges of the deprecated feature and
the privileges of the features that are supposed to replace them is
expressed with a new `replacedBy` property available on the privileges
of the deprecated feature.

<details>

<summary>Click to see the code</summary>

```ts
deps.features.registerKibanaFeature({
  // This is a new `KibanaFeature` property available during feature registration.
  deprecated: {
    // User-facing justification for privilege deprecation that we can display
    // to the user when we ask them to perform role migration.
    notice: i18n.translate('xpack.security...', {
      defaultMessage: "Feature Alpha is deprecated, refer to {link}...",
      values: { link: docLinks.links.security.deprecatedFeatureAlpha },
    })
  },
  // Feature id should stay unchanged, and it's not possible to reuse it.
  id: 'feature_alpha',
  name: 'Feature Alpha (DEPRECATED)',
  privileges: {
    all: {
      savedObject: { all: ['saved_object_1', 'saved_object_2'], read: [] },
      ui: ['ui_all'],
      api: ['api_all'],
      replacedBy: [
        { feature: 'feature_beta', privileges: ['all'] },
        { feature: 'feature_gamma', privileges: ['all'] },
      ],
      … omitted for brevity …
    },
    read: {
      savedObject: { all: [], read: ['saved_object_1', 'saved_object_2'] },
      ui: ['ui_read'],
      api: ['api_read'],
      replacedBy: [
        { feature: 'feature_beta', privileges: ['read'] },
        { feature: 'feature_gamma', privileges: ['read'] },
	],
      … omitted for brevity …
    },
  },
  … omitted for brevity …
});
```

</details>

### Step 4: Adjust the code to rely only on new, non-deprecated features

Special care should be taken if the replacement privileges cannot reuse
the API access tags from the deprecated privileges and introduce new
tags that will be applied to the same API endpoints. In this case,
developers should replace the API access tags of the deprecated
privileges with the corresponding tags provided by the replacement
privileges. This is necessary because API endpoints can only be accessed
if the user privileges cover all the tags listed in the API endpoint
definition, and without these changes, existing roles referencing
deprecated privileges won’t be able to access those endpoints.

The UI capabilities are handled slightly differently because they are
always prefixed with the feature ID. When migrating to new features with
new IDs, the code that interacts with UI capabilities will be updated to
use these new feature IDs.

<details>

<summary>Click to see the code</summary>

```ts
// BEFORE deprecation/migration
// 1. Feature Alpha defition (not deprecated yet)
deps.features.registerKibanaFeature({
  id: 'feature_alpha',
  privileges: {
    all: {
      api: ['api_all'],
      … omitted for brevity …
    },
  },
  … omitted for brevity …
});

// 2. Route protected by `all` privilege of the Feature Alpha
router.post(
  { path: '/api/domain/my_api', options: { tags: ['access:api_all'] } },
  async (_context, request, response) => {}
);

// AFTER deprecation/migration
// 1. Feature Alpha defition (deprecated, with updated API tags)
deps.features.registerKibanaFeature({
  deprecated: …,
  id: 'feature_alpha',
  privileges: {
    all: {
      api: ['api_all_v2'],
      replacedBy: [
        { feature: 'feature_beta', privileges: ['all'] },
      ],
      … omitted for brevity …
    },
  },
  … omitted for brevity …
});

// 2. Feature Beta defition (new)
deps.features.registerKibanaFeature({
  id: 'feature_beta',
  privileges: {
    all: {
      api: ['api_all_v2'],
      … omitted for brevity …
    }
  },
  … omitted for brevity …
});

// 3. Route protected by `all` privilege of the Feature Alpha OR Feature Beta
router.post(
  { path: '/api/domain/my_api', options: { tags: ['access:api_all_v2'] } },
  async (_context, request, response) => {}
);

----

//  Old client-side code (supports only deprecated privileges)
if (capabilities.feature_alpha.ui_all) {
  … omitted for brevity …
}

//  New client-side code (will work for **both** new and deprecated privileges)
if (capabilities.feature_beta.ui_all) {
  … omitted for brevity …
}
```
</details>

## How to test

The code introduces a set of API integration tests that are designed to
validate whether the privilege mapping between deprecated and
replacement privileges maintains backward compatibility.

You can run the test server with the following config to register a
number of [example deprecated
features](https://github.com/elastic/kibana/pull/186800/files#diff-d887981d43bbe30cda039340b906b0fa7649ba80230be4de8eda326036f10f6fR20-R49)(`x-pack/test/security_api_integration/plugins/features_provider/server/index.ts`)
and the features that replace them, to see the framework in action:

```bash
node scripts/functional_tests_server.js --config x-pack/test/security_api_integration/features.config.ts
```

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-10-14 14:40:59 -05:00
Stephen Schmidt
7312155fc1
chore(quality-gates): update manual judgement on staging QGs (#196165)
This is a build/promotion system change only, there is no code related
changes here.

- Similar to https://github.com/elastic/kibana/pull/195944, but for the
non-emergency pipeline in staging
- For now, only run the staging manual judgement on slice:
'staging-ds-2'
- Since slices can be grouped as a comma delimited list, we're choosing
to be overly cautious with the regex
2024-10-14 12:24:04 -05:00
Irene Blanco
96966c5113
[ECO][Infra] Add callout for ingesting metrics data in Host and Container views (#195378)
## Summary

Closes https://github.com/elastic/kibana/issues/193703

This PR introduces a callout designed to prompt users to ingest metrics
data in the Host and Container views.
The callout will be displayed on the following tabs:

- **Hosts**: Overview, Metrics, Processes
- **Containers**: Overview, Metrics

The primary condition for showing the callout is that the asset does not
currently have any metrics data available. This enhancement aims to
encourage users to take action and improve their monitoring experience.

Additional details include:

- The callout will be positioned below the date picker for better
visibility.
- Links for "Add Metrics" will guide users to the appropriate onboarding
pages based on their asset type.
- The callout will be dismissible on the Overview tab, and the KPI
section will be hidden in favor of the callout for a cleaner interface.
- Custom telemetry events will be tracked to measure user interactions
with the callout.
- Only Docker and K8 containers will show the callout.

**Host**
|Tab||
|-|-|
|Overview|![Screenshot 2024-10-08 at 12 19
22](https://github.com/user-attachments/assets/e357d6c6-2423-40f9-a513-361c642dc07c)|
|Metrics|![Screenshot 2024-10-08 at 12 19
31](https://github.com/user-attachments/assets/559a6e71-344a-4b4a-9ad6-8d229a1d9bcb)|
|Processes|![Screenshot 2024-10-08 at 12 19
39](https://github.com/user-attachments/assets/070f6fb1-0756-4b21-abce-4b395be943df)|

**Container**
|Tab||
|-|-|
|Overview|![Screenshot 2024-10-08 at 12 24
10](https://github.com/user-attachments/assets/101cfc7b-f445-44e7-9aa3-bec8928c3ed5)|
|Metrics|![Screenshot 2024-10-08 at 12 21
22](https://github.com/user-attachments/assets/d516d449-2af4-441f-9047-39c9362c5a86)|

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Caue Marcondes <caue.marcondes@elastic.co>
2024-10-14 16:42:44 +02:00
Stephen Schmidt
95b0747ee7
chore(quality-gates): update manual judgement on staging QGs (#195944)
This is a build/promotion system change only, there is no code related
changes here.

- For now, only run the staging manual judgement on slice:
'staging-ds-2'
- Since slices can be grouped as a comma delimited list, we're choosing
to be overly cautious with the regex
2024-10-11 17:14:53 -05:00
Brandon Kobel
e7720b80aa
Limiting prod quality-gate SLOs to tag "kbn-quality-gate" (#195578)
Without this change, the quality gates check for production promotions
include all SLOs with the kibana tag. This results in us checking a ton
of duplicate SLOs because we have per-orchestrator and per-project-type
SLOs.


Example of a buildkite run with the new tag:
https://buildkite.com/elastic/serverless-quality-gates/builds/23385#019276f7-dccd-422e-a75a-66140e3232c7
2024-10-11 15:02:34 -04:00
Jon
d1babbe133
[ci] Revert org-wide commit status checks (#195954) 2024-10-11 10:51:24 -05:00
Alex Szabo
5b43c1f0f0
update triggered pipeline name as well (#195894)
## Summary
We've added a new pipeline in #195581, where the name update wasn't
synced to the PR configuration, so the new button is looking for a
non-existent pipeline.

This PR updates the target
2024-10-11 16:28:46 +02:00
Robert Oskamp
e33de6d6ae
Revert "skip failing configs (#195811)" (#195890)
## Summary

This PR re-enables the serverless deployment agnostic tests

### Details

- The root cause of the problem should be fixed with #195563
- This reverts commit 8d1bc50335

Closes #195811
2024-10-11 15:52:44 +02:00
Alex Szabo
b05a3750ad
[ci] Fix bad commit (#195892)
## Summary
Reverts #195891 , fixes prConfig type issue coming from #195581
2024-10-11 13:19:34 +02:00
Alex Szabo
bde4064ba0
fix bad commit (#195891)
## Summary
TODO
2024-10-11 13:14:24 +02:00
Jon
57ce3802de
[ci] Update verify_es_serverless_image.yml patterns (#195813)
To make sure these test suites are run during elasticsearch image
verification.

Related to https://github.com/elastic/kibana/issues/195811
2024-10-11 12:18:37 +02:00
Alex Szabo
fb0a9334f0
[ci] Click to deploy project (#195581)
## Summary
There's a use-case where developers would like to work while testing
their deployments in a serverless environment, or they'd like to
redeploy only, without running the whole suite of tests before. Labels
are still required because project deployment requires the project kind
to be prescribed.

This PR adds the instruments to do so:
 - pipeline resource definition to create a new pipeline
 - pipeline implementation, that only builds and deploys
- `pull-requests.json` configuration, so the option shows up in the
'click to xy' section

With this, we present a cheaper, faster way to deploy serverless when
needed.

Closes: https://github.com/elastic/kibana-operations/issues/121

---------

Co-authored-by: Brad White <Ikuni17@users.noreply.github.com>
2024-10-11 11:16:54 +02:00
Nicolas Chaulet
5b69749997
[Fleet] Add cypress test against space awareness (#195372) 2024-10-11 09:19:21 +02:00
Brad White
56b478fef2
[CI] Skip ci for devcontainer changes (#195814)
## Summary

For now it is not necessary to run any of CI for `.devcontainer`
changes.
2024-10-10 16:48:45 -07:00
Jonathan Budzenski
8d1bc50335 skip failing configs (#195811) 2024-10-10 12:44:26 -05:00
Karen Grigoryan
bc75d03f5c
ci(deploy): dead deploy fix script (#195753)
This fixes edge case with dead deploys failing current server deploy
jobs in https://github.com/elastic/kibana/pull/191898
2024-10-10 17:29:49 +02:00
Jon
d051743e6b
[ci] Rebuild image after elasticsearch promotion (#195671)
1) After an elasticsearch image is promoted, this triggers a VM rebuild
to update the snapshot cache
1) Moves elasticsearch builds to later in the day, when there's less
activity.
2024-10-09 18:29:09 -05:00
Robert Oskamp
554ec2e321
Tag pipelines related to Kibana serverless release (#195631)
## Summary

This PR tags all pipelines that are related to the Kibana serverless
release (including requirements like on-merge and artifacts build) with
`kibana-serverless-release`. This will allow us to easily find these
pipelines in Buildkite.
2024-10-09 22:28:19 +02:00
Rodney Norris
71fd96ad8b
[Search][FTR] Solution Nav (#195327)
## Summary

Adding `functional_search` suite with a set of test for the search
solution navigation. But this suite will also grow to test search
solution pages that do not require the enterprise search node.

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
2024-10-09 08:45:24 -05:00
Alex Szabo
94934505eb
[CI] Add tags for kibana pipelines (#195388)
## Summary
Kibana-related pipelines are hard to find on Buildkite, due to other,
ingest-related pipelines having 'kibana' in their names.

This pipeline adds tags to pipelines serving `kibana` CI duties, so they
can be easily found using Buildkite's tags/labels.

The tags added are mostly `kibana` but some pipelines also got the
`security-solution` label, as these pipelines can be easily associated
with the served solution.
2024-10-09 11:26:46 +02:00
Mario Duarte
942a1f1e69
CP-7782 - Replace E2E pipeline execution with serverless-quality-gates CHECK_SYNTHETICS (#195214) 2024-10-08 21:57:54 +02:00
Alex Szabo
fa42f06b8d
[ci] Fix chrome forward testing pipeline (#195119)
## Summary
Added in #194682, this pipeline is failing, because we were reusing the
PR pipeline, that at some parts expect PR-related env-vars to be around.

This PR introduces a new pipeline implementation, so we're no longer
sharing the pipeline with the PR jobs. This prevents errors coming from
assuming there are relative changes in the context.

The PR also propagates the `USE_CHROME_BETA` flag to the cypress runners
of the security solution scripts (I've noticed most other pipelines rely
on these runners).

This run demonstrates the pipeline:
https://buildkite.com/elastic/kibana-chrome-forward-testing/builds/9
2024-10-08 12:31:43 +02:00
Jon
65f634523e
Update forward testing pipeline to report test failures (#195251)
Also snuck in a commit to update a lingering SLACK_NOTIFICATIONS_ENABLED
to ELASTIC_SLACK_NOTIFICATIONS_ENABLED
2024-10-07 19:29:33 -05:00
Jon
149c514e83
Re-enable org wide PR bot (#195131)
https://github.com/elastic/kibana/pull/194768 without the merge
conflicts.

Switches over to the org wide PR bot, with backwards compatibility for
both versions.

Updating the pipeline definition here is a global set for environment
variables on all branches, so I intend on merging the backports first to
support both versions and then proceeding with this.
2024-10-07 13:00:12 -05:00
Julia Bardi
52abebf0cb
[Fleet] remove old bundled.yaml from oas, fixed tags (#194788)
## Summary

Closes https://github.com/elastic/kibana/issues/184685

**Release notes**: These schema changes shouldn't be breaking, but there
were some incorrect/missing response schemas in the old openapi spec.

For example the API `POST /api/fleet/agents/{agentId}/actions` response
was incorrectly documented:
https://petstore.swagger.io/?url=https://raw.githubusercontent.com/elastic/kibana/main/x-pack/plugins/fleet/common/openapi/bundled.json#/Elastic%20Agent%20actions/new-agent-action

```
{
  "body": [
    0
  ],
  "statusCode": 0,
  "headers": "string"
}
```
Fixed here:
31f8cfd6ef/oas_docs/bundle.json#/Elastic%20Agent%20actions/%252Fapi%252Ffleet%252Fagents%252F%257BagentId%257D%252Factions%230
```
{
  "item": {
    "ack_data": "string",
    "agents": [
      "string"
    ],
    "created_at": "string",
    "data": "string",
    "expiration": "string",
    "id": "string",
    "minimum_execution_duration": 0,
    "namespaces": [
      "string"
    ],
    "rollout_duration_seconds": 0,
    "sent_at": "string",
    "source_uri": "string",
    "start_time": "string",
    "total": 0,
    "type": "string"
  }
}
```

The new spec should match the implementation accurately, and responses
are being verified when returned. Tests were added to make sure the
response schemas are correct.
If there are any bugs in the current schema, it will result in a HTTP
500 error with an error message on where the schema validation failed.
Example of an error where a field is missing:
```
{
    "statusCode": 500,
    "error": "Internal Server Error",
    "message": "Failed output validation: [request body.items.0.name]: definition for this key is missing"
}
```
Example of an error where a field is mandatory in the schema, but not
provided in the response (missing `schema.maybe`)
```
{
    "statusCode": 500,
    "error": "Internal Server Error",
    "message": "Failed output validation: [request body.items.0.internal]: expected value of type [boolean] but got [undefined]"
}
```

There are a few places where the validation allows unknown fields. Used
it where some fields were not included in TS types or fields are more
dynamic, e.g. fields coming from packages or elasticsearch settings.

https://github.com/search?q=repo%3Aelastic%2Fkibana+extendsDeep+path%3A%2F%5Ex-pack%5C%2Fplugins%5C%2Ffleet%5C%2Fserver%5C%2Ftypes%5C%2F%2F&type=code
```
.extendsDeep({
  unknowns: 'allow',
 })
```

Changes in this pr:
Remove using old `bundled.yaml` to generate oas, fixed tags.
Removed old openapi files, updated readme.

Here is the new bundle in Swagger UI: 

[stateful](31f8cfd6ef/oas_docs/bundle.json)

[serverless](da72ee0093/oas_docs/bundle.serverless.json)

Updated serverless scripts too.

Updated Fleet readme:
da72ee0093/x-pack/plugins/fleet/common/openapi/README.md

Generated the new bundle by running this script locally:
```
node scripts/capture_oas_snapshot --include-path /api/fleet --update
```

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-10-07 11:00:12 +02:00
Alex Szabo
1fb52db808
[ci] Create pipeline for chrome forward-testing against beta versions (#194682)
## Summary
Creates a pipeline for testing against `google-chrome-beta`.

- Uses the flag: `USE_CHROME_BETA` to switch functional tests to running
against chrome-beta.
- Uses the same pipeline file PR builds are using
- Only FTRs will be using `google-chrome-beta`, so other test types are
disabled
- [x] check if other test types (integration / cypress) can be directed
to `google-chrome-beta`.
  
  
Tested with the migration staging pipeline:
https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/164

Connected to: https://github.com/elastic/kibana-operations/issues/199
2024-10-04 17:26:35 +02:00
Jon
cef36d130a
Revert "[ci] Split artifacts-container-image alerts (#194706)" (#194837)
This reverts commit 8cbd641239.

No luck, this didn't work as I had hoped. Will have to follow up with
something in the build bot.
2024-10-03 16:08:28 -07:00
Jeramy Soucy
26f2928b08
Set spaces and roles CRUD APIs to public (#193534)
Closes #192153

## Summary

This PR sets the spaces and roles CRUD operation HTTP API endpoints to
public in both stateful and serverless offerings, and additionally,
switches to the versioned router to register these endpoints.

Prior to this PR, the access level was not explicitly set, thus any
endpoints registered in serverless were by default internal. CRUD
operations for spaces and roles are being set to public to support the
rollout of custom roles in serverless, which coincides with enabling
multiple spaces.

### Note
- Currently, roles APIs are only available in serverless via a feature
flag (`xpack.security.roleManagementEnabled`)
- Spaces APIs are already registered in serverless, however, the maximum
number of spaces is by default 1, rendering create and delete operations
unusable. By overriding `xpack.spaces.maxSpaces` to a number greater
than 1 (stateful default is 1000), it will effectively enable use of the
spaces CRUD operations in serverless.

## Tests
-
x-pack/test_serverless/api_integration/test_suites/common/management/multiple_spaces_enabled.ts
-
x-pack/test_serverless/api_integration/test_suites/common/management/spaces.ts
-
x-pack/test_serverless/api_integration/test_suites/common/platform_security/authorization.ts
-
x-pack/test_serverless/api_integration/test_suites/common/platform_security/roles_routes_feature_flag.ts
- Unit tests for each endpoint (to account for versioned router)
- Flaky Test Runner:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7002

## Manual Testing
1. Start ES & Kibana in serverless mode with config options to enable
role management and multiple spaces

Elasticsearch:
```
xpack.security.authc.native_roles.enabled: true
```
 KIbana:
```
 xpack.security.roleManagementEnabled: true
 xpack.spaces.maxSpaces: 100
```
3. Issue each CRUD HTTP API without including the internal origin header
('x-elastic-internal-origin') and verify you do not receive a 400 with
the message "method [get|post|put|delete] exists but is not available
with the current configuration"
4. Repeat steps 1 & 2 from the current head of main and verify that you
DO receive a 400 with the message "method [get|post|put|delete] exists
but is not available with the current configuration"

Regression testing - ensure that interfaces which leverage spaces and
roles APIs are functioning properly
- Spaces management
- Space navigation
- Roles management

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-10-03 16:28:54 +02:00
Anton Dosov
568e40acca
Clean up cloud_chat (#194571)
## Summary

Close https://github.com/elastic/kibana-team/issues/1017

This PR removes the unused Cloud Chat functionality from Kibana. The
chat was not used for some time. Moreover, we've seen some issues with
it where users saw it when it wasn't expected. Given the absence of
automated tests and the fact that the feature is no longer needed, we
are removing it to improve the overall maintainability and reliability
of the codebase. This will also decrease the amount of code loaded for
trial users of Kibana in cloud making the app slightly faster.
2024-10-03 13:37:47 +02:00
Jon
8cbd641239
[ci] Split artifacts-container-image alerts (#194706)
This modifies alerts to route build failures to our general alert
channel and leaves test failures to the test alerts channel
2024-10-02 15:23:18 -05:00
Sandra G
6ed731cad5
[Obs AI Assistant] Serverless API integration tests (#192219)
Tests for serverless

- copies over and modifies all tests from stateful to work in
serverless. ~~deployment agnostic tests do not yet support enterprise
license for stateful, so are tests don't yet qualify as being deployment
agnostic~~. Given how difficult it is to see differences from the
stateful tests, I've added PR comments where I've changed something that
might be of interest.
- changes to `createObservabilityAIAssistantApiClient` to use supertest
without basic auth and accept headers for serverless and use roles
- removes creating persisted users when tests start and [use
roles](https://github.com/elastic/kibana/blob/main/x-pack/test_serverless/README.md#roles-based-testing)
within tests. its not possible to create custom users with the
serverless test framework at the moment. See
https://github.com/elastic/kibana/issues/192711

Skipped tests
- knowledge base tests https://github.com/elastic/kibana/issues/192886
- any test suite that uses the LLM proxy has been skipped on MKI
https://github.com/elastic/kibana/issues/192751
- all tests that depend on the config.modelId skipped in MKI
https://github.com/elastic/kibana/issues/192757

TODO:

- [x] move over remaining tests
- [x]  test in MKI environment before merging
- [x] create issues for skipped tests
- [ ] this will not run on MKI (after merging) unless we ping the
appex-qa team to add it to the pipeline. this is due to creating a
separate config. ask appex-qa team to add our config.


Followup / related issues to be tracked in a newly created issue:

- [ ] https://github.com/elastic/kibana/issues/192757
- [ ] https://github.com/elastic/kibana/issues/192886
- [ ] https://github.com/elastic/kibana/issues/192751
- [ ] https://github.com/elastic/kibana/issues/192701
- [ ] https://github.com/elastic/kibana/issues/192497
- [ ] https://github.com/elastic/kibana/issues/192711
- [ ] https://github.com/elastic/kibana/issues/192718
- [ ] serverless functional tests
- [ ] inquire with ml-ui-team to have the ability to delete system
indices which we do after uninstalling tiny elser with .ml indices
2024-09-30 19:07:01 -04:00
Jon
0dada14ac5
[cloud deploy] Fix deployment config (#194076)
Followup to https://github.com/elastic/kibana/pull/193101

Fixes
https://buildkite.com/elastic/kibana-pull-request/builds/237163#01922ad1-38ef-48e6-a0bd-d3f051c4bb0f/537-542
2024-09-30 09:20:16 -05:00
Stratoula Kalafateli
64a0bdc1d6
[Web logs journey] Replaces the rest visualizations with Lens charts (#194097)
## Summary

Updates the web logs journey to use Lens charts instead of deprecated
ones. Specifically there was one TSVB and one goal that got replaced.

Also there were 3 legacy metric Lens visualizations. I updated them to
use the lens metric type
2024-09-26 17:41:16 +02:00
Mario Duarte
5e2a575a08
CP-7781 - switching bk pipeline to gpctl promote after synthetics (#193411) 2024-09-25 09:49:24 -05:00
Stratoula Kalafateli
d56a1bbbba
[ES|QL] Renames the textbased editor to esql editor (#193521)
## Summary

Renames the text-based-editor to esql-editor

I tried to also rename components, data-test-subj, classNames and files.
My focus is mostly on the plugin and package of the esql editor

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-09-23 12:06:53 +02:00
Stratoula Kalafateli
9d0d41ce87
[ES|QL] Renames the language package to a more generic name (#193517)
## Summary

Renames the language-documentation-popover to language-documentation as
it stores additional layouts:
- popover
- flyout
- inline
2024-09-23 09:58:53 +02:00
Maxim Palenov
a378064006
[HTTP/OAS] Auto bundle staging OpenAPI specs (#193402)
**Relates to:** https://github.com/elastic/kibana/pull/189262

## Summary

This PR adds scripts to automatically bundle staging Kibana OpenAPI specs in CI. It's done in a similar way as in https://github.com/elastic/kibana/pull/189262.
2024-09-22 09:15:57 +02:00
Stratoula Kalafateli
44adb7353d
[ES|QL] Performance journey for dashboards made with ES|QL (#193322)
## Summary

Closes https://github.com/elastic/kibana/issues/182548

Creates an dashboard journey similar to the web logs one but the
majority of the visualizations are using ES|QL.

Note: There is one Lens viz not converted yet because it needs
inlinestats. We could also convert it when inlinestats go to GA.
2024-09-19 17:57:44 +02:00
Stratoula Kalafateli
dfbd7de3f5
[ES|QL] Enhances the inline documentation experience (#192156)
## Summary

Closes https://github.com/elastic/kibana/issues/166907

This PR:

1. Changes the inline docs implementation to a flyout for Discover

<img width="1256" alt="image"
src="https://github.com/user-attachments/assets/3665c73a-82d5-49b9-88c3-e129eda63885">

2. Adds the flyout to open from the help menu and removes it from the
editor footer

3. For the inline editing changes the docs to be rendered as a component
inside the editor (exactly as the history component)

Note: The documentation in the inline editing has limited space (same
problem has the history component). I need to sync with Ryan to see what
we can do, I am exploring some ideas in follow up PRs

<img width="647" alt="image"
src="https://github.com/user-attachments/assets/2fa9fcc3-2a07-4bea-b675-5ae6578696d3">

4. Moves the esql docs to the language package and the script associated
to this

### Follow up
- The language package should be renamed as the `popover` makes sense
only for the formula implementation now. I will do it after this PR gets
merged
- See how we can give more space to the docs / history container when in
inline mode

### Note
The async bundle got increased as we want the ES|QL docs in unified
search too. It wont be a problem as is loaded asynchronously.

### Checklist

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Ryan Keairns <contactryank@gmail.com>
Co-authored-by: Drew Tate <andrew.tate@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2024-09-19 17:10:39 +02:00
Pablo Machado
27f5da436b
[SecuritySolutions] Create Entity Store 'entities/list' API (#192806)
This PR introduces the following API routes for listing Entity Store
"entities":

<meta charset="utf-8"><b style="font-weight:normal;"
id="docs-internal-guid-9410c5d7-7fff-e873-6830-887939a306fb"><div
dir="ltr" style="margin-left:-0.75pt;" align="left">
List Entities | GET /api/entity_store/entities/list
-- | --
</div></b>

The PR includes the following:
 - The OpenAPI schemas for the route
 - The actual Kibana side endpoint
 - Add searchEntities function to the `EntityStoreDataClient`
 

### How to test

1. Add some host/user data
* Easiest is to use
[elastic/security-data-generator](https://github.com/elastic/security-documents-generator)
2. Make sure to add `entityStoreEnabled` under
`xpack.securitySolution.enableExperimental` in your `kibana.dev.yml`
3. In kibana dev tools or your terminal, call the `INIT` route for
either `user` or `host`.
4. You should now see 2 transforms in kibana. Make sure to re-trigger
them if needed so they process the documents.
5. Call the new API, and it should return entities 



Implements https://github.com/elastic/security-team/issues/10517

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-09-19 07:54:53 -05:00
Elena Shostak
28aa274f66
Updated js-yaml to v4 (#190678)
## Summary
Updated `js-yaml` to `4.1.0`.

This PR also introduces a type override for the `js-yaml` load function
to maintain compatibility across the codebase. Specifically, updated
type definition of the load function looks as follows:

```typescript
function load<T = any>(str: string, opts?: jsyaml.LoadOptions): T;
```

The original type definition of the load function in `js-yaml` changed
from `any` to `unknown`. This change would require extensive type
updates throughout the entire repository to accommodate the `unknown`
type. To avoid widespread type changes and potential issues in the
codebase, the type is overriden back to `any` for now.
This is a temporary measure, we plan to address the necessary type
changes in subsequent PRs, where teams will gradually update the
codebase to work with the `unknown` type.


### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

## Release note
Updated `js-yaml` to `4.1.0`.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co>
2024-09-19 12:25:03 +02:00
Dzmitry Lemechko
f5975d28fa
[performance] support triggering subset of journeys against KIbana PR in CI (#193175)
## Summary

It’s common request for Dev teams to run specific journeys on a PR to
compare performance metrics against the `main` branch. These requests
usually focus on a particular area, such as the Dashboard or Discover
app.

To streamline the process, this PR groups relevant journeys into
categories that can be triggered through an environment variable. For
example, setting `JOURNEYS_GROUP=dashboard` will execute only the three
dashboard-specific journeys, which are (usually) sufficient for
evaluating the performance impact of code changes within the Dashboard
app.

Current Process for Triggering Performance Builds:
- Create a new kibana-single-user-performance
[build](https://buildkite.com/elastic/kibana-single-user-performance#new)
- Provide the following arguments:

Branch: `refs/pull/<PR_number>/head`
Under Options, set the environment variable:
`JOURNEYS_GROUP=<group_name>`

Currently supported journey groups:
- kibanaStartAndLoad
- crud
- dashboard
- discover
- maps
- ml

[Build example

](https://buildkite.com/elastic/kibana-single-user-performance/builds/14427)
Each group focuses on a specific set of journeys tied to its respective
area in Kibana, allowing for more targeted performance testing. Since
running group takes ~5-10 min on bare metal worker, it should not delay
the regular (every 3h) runs against `main` branch


test locally with `node scripts/run_performance.js --group <group_name>`
2024-09-18 14:53:02 +02:00
Hannah Mudge
5082eef2f1
[Embeddable Rebuild] [Controls] Remove non-React controls from controls plugin (#192017)
Part of https://github.com/elastic/kibana/issues/192005
Closes https://github.com/elastic/kibana/issues/176533

## Summary

This PR represents the first major cleanup task for the control group
embeddable refactor. The tasks included in this PR can be loosely
summarized as follows:
1. This PR removes the old, non-React version of controls 
- Note that the new controls are still included under the
`react_controls` folder - I will address this in a follow up PR.
2. This PR removes **all** types associated with the old embeddable
system; i.e. any `*input*` or `*output*` types.
- As part of cleaning up these types, some of the types included in the
`public/react_controls` folder had to be moved to `common` to make them
available to server-side code.
- This resulted in an... unfortunate number of import changes 🫠 Hence
the rather large file change count. I took this opportunity to organize
the imports, too - so a significant chunk of these files are simply
import changes.
3. This PR removes the controls Storybook and all related mocks
- Since the controls storybooks have been broken for awhile, and since
we had plans to remove them but never got around to it, I just decided
to delete them as part of this PR and close
https://github.com/elastic/kibana/issues/176533 rather than spending
time to fix the types for non-operational stories

### Checklist

- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios


### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-09-17 08:12:54 -06:00
Ignacio Rivas
b3a1e5fb8f
[Console] UX Improvements for phase 2 (#190698) 2024-09-17 16:35:20 +03:00
Alex Szabo
e66450ea4c
[CI] Run PR against chrome-beta (#192257)
## Summary
This PR's goal is to enable developers (or automation) to test against
chrome-beta with a PR label - to adjust sensitive tests, or to foresee
test breakages.

⚠️ Risk: a PR verified/built with the chrome beta setting, would pass
the PR tests, but might not pass the on-merge suite in the end.

Addresses: https://github.com/elastic/kibana-operations/issues/199
Depends on: https://github.com/elastic/ci-agent-images/pull/907

It highlights the errors visible only on the next versions:
https://buildkite.com/elastic/kibana-pull-request/builds/233373
And it doesn't break with non-beta run:
https://buildkite.com/elastic/kibana-pull-request/builds/233716
2024-09-17 12:41:33 +02:00
Jon
c5c79a954c
[ci/cloud deploy] Remove enterprise search (#193101) 2024-09-16 18:14:12 -05:00