## Summary
`get_all` suite
`x-pack/test/spaces_api_integration/security_and_spaces/apis/get_all.ts`
is intented to be run only with `basic` license, since FIPS overrides it
we need to skip that test for FIPS.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
Epic: https://github.com/elastic/security-team/issues/7998
In this PR we're breaking out the `timeline` and `notes` features into
their own feature privilege definition. Previously, access to both
features was granted implicitly through the `siem` feature. However, we
found that this level of access control is not sufficient for all
clients who wanted a more fine-grained way to grant access to parts of
security solution.
In order to break out `timeline` and `notes` from `siem`, we had to
deprecate it feature privilege definition for. That is why you'll find
plenty of changes of `siem` to `siemV2` in this PR. We're making use of
the feature privilege's `replacedBy` functionality, allowing for a
seamless migration of deprecated roles.
This means that roles that previously granted `siem.all` are now granted
`siemV2.all`, `timeline.all` and `notes.all` (same for `*.read`).
Existing users are not impacted and should all still have the correct
access. We added tests to make sure this is working as expected.
Alongside the `ui` privileges, this PR also adds dedicated API tags.
Those tags haven been added to the new and previous version of the
privilege definitions to allow for a clean migration:
```mermaid
flowchart LR
subgraph v1
A(siem) --> Y(all)
A --> X(read)
Y -->|api| W(timeline_write / timeline_read / notes_read / notes_write)
X -->|api| V(timeline_read /notes_read)
end
subgraph v2
A-->|replacedBy| C[siemV2]
A-->|replacedBy| E[timeline]
A-->|replacedBy| G[notes]
E --> L(all)
E --> M(read)
L -->|api| N(timeline_write / timeline_read)
M -->|api| P(timeline_read)
G --> Q(all)
G --> I(read)
Q -->|api| R(notes_write / notes_read)
I -->|api| S(notes_read)
end
```
### Visual changes
#### Hidden/disabled elements
Most of the changes are happening "under" the hood and are only
expressed in case a user has a role with `timeline.none` or
`notes.none`. This would hide and/or disable elements that would usually
allow them to interact with either timeline or the notes feature (within
timeline or the event flyout currently).
As an example, this is how the hover actions look for a user with and
without timeline access:
| With timeline access | Without timeline access |
| --- | --- |
| <img width="616" alt="Screenshot 2024-12-18 at 17 22 49"
src="https://github.com/user-attachments/assets/a767fbb5-49c8-422a-817e-23e7fe1f0042"
/> | <img width="724" alt="Screenshot 2024-12-18 at 17 23 29"
src="https://github.com/user-attachments/assets/3490306a-d1c3-41aa-af5b-05a1dd804b47"
/> |
#### Roles
Another visible change of this PR is the addition of `Timeline` and
`Notes` in the edit-role screen:
| Before | After |
| ------- | ------ |
| <img width="746" alt="Screenshot 2024-12-12 at 16 31 43"
src="https://github.com/user-attachments/assets/20a80dd4-c214-48a5-8c6e-3dc19c0cbc43"
/> | <img width="738" alt="Screenshot 2024-12-12 at 16 32 53"
src="https://github.com/user-attachments/assets/afb1eab4-1729-4c4e-9f51-fddabc32b1dd"
/> |
We made sure that for migrated roles that hard `security.all` selected,
this screen correctly shows `security.all`, `timeline.all` and
`notes.all` after the privilege migration.
#### Timeline toast
There are tons of places in security solution where `Investigate / Add
to timeline` are shown. We did our best to disable all of these actions
but there is no guarantee that this PR catches all the places where we
link to timeline (actions). One layer of extra protection is that the
API endpoints don't give access to timelines to users without the
correct privileges. Another one is a Redux middleware that makes sure
timelines cannot be shown in missed cases. The following toast will be
shown instead of the timeline:
<img width="354" alt="Screenshot 2024-12-19 at 10 34 23"
src="https://github.com/user-attachments/assets/1304005e-2753-4268-b6e7-bd7e22d8a1e3"
/>
### Changes to predefined security roles
All predefined security roles have been updated to grant the new
privileges (in ESS and serverless). In accordance with the migration,
all roles with `siem.all` have been assigned `siemV2.all`,
`timeline.all` and `notes.all` (and `*.read` respectively).
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: PhilippeOberti <philippe.oberti@elastic.co>
Co-authored-by: Steph Milovic <stephanie.milovic@elastic.co>
## Summary
Moved Search Applications & Behavioral Analytics from the base
KibanaFeature to their own so that they can be controlled independently
from other parts of the search solution.
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
This PR adds:
* Some basic features and privileges for the EEM app
* A function that sets up an index with a template for the new
definitions
* 4 API endpoints to read and create entity types and sources
* `POST /internal/entities/v2/definitions/types`
* `GET /internal/entities/v2/definitions/types`
* `POST /internal/entities/v2/definitions/sources`
* `GET /internal/entities/v2/definitions/sources`
* Some v2 shuffling around of code
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co>
## Summary
This pr adds 2 new sub feature permissions to the cases plugin in
stack/security/observability, that behave as follows. The first is for
controlling the ability to reopen cases. When Cases has the read
permission, and the reopen permission is not enabled, users have
permissions as before. When enabled, users can move cases from closed to
open/in progress, but nothing else. If a user has all and this
permission, they can do anything as before, if the option is unselected,
they can change case properties, and change a case from open to
anything, in progress to anything, but if the case is closed, are unable
to reopen it.
The 2nd permission is 'Add comment'. When enabled and the user has case
read permissions, users can add comments, but not make any other changes
to the case. When the user has read and this deselected, read functions
as before. When a user has this permission and cases is all, this
functions as all. When they have all but this permission is deselected,
the user can do everything normally, except add cases comments.
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Michael Olorunnisola <michael.olorunnisola@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
### Authz API migration for authorized routes
This PR migrates `access:<privilege>` tags used in route definitions to
new security configuration.
Please refer to the documentation for more information: [Authorization
API](https://docs.elastic.dev/kibana-dev-docs/key-concepts/security-api-authorization)
### **Before migration:**
Access control tags were defined in the `options` object of the route:
```ts
router.get({
path: '/api/path',
options: {
tags: ['access:<privilege_1>', 'access:<privilege_2>'],
},
...
}, handler);
```
### **After migration:**
Tags have been replaced with the more robust
`security.authz.requiredPrivileges` field under `security`:
```ts
router.get({
path: '/api/path',
security: {
authz: {
requiredPrivileges: ['<privilege_1>', '<privilege_2>'],
},
},
...
}, handler);
```
### What to do next?
1. Review the changes in this PR.
2. You might need to update your tests to reflect the new security
configuration:
- If you have tests that rely on checking `access` tags.
- If you have snapshot tests that include the route definition.
- If you have FTR tests that rely on checking unauthorized error
message. The error message changed to also include missing privileges.
## Any questions?
If you have any questions or need help with API authorization, please
reach out to the `@elastic/kibana-security` team.
---------
Co-authored-by: Elena Shostak <elena.shostak@elastic.co>
closes [#3874](https://github.com/elastic/observability-dev/issues/3874)
## 📝 Summary
This PR adds new kibana privilege feature to control access to `Data Set
Quality` page under Stack Management's `Data` section.
Had to fix a lot of tests since the `kibana_admin` role gets access by
default to all kibana features one of which now is the `Data Set
Quality` page. At the same time this made the `Data` section visible to
any user with `kibana_admin` role.
## 🎥 Demo
https://github.com/user-attachments/assets/ce8c8110-f6f4-44b8-a4e7-5f2dd3deda66
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This implements RBAC properly for Search. We no longer rely on the
enterprise search node to limit access to Search, so except for App
Search and Workplace Search. That means we need proper Kibana features
to manage RBAC, which this adds: one for Search and a separate one for
Relevance as that feature is guarded by an Enterprise license.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Kibana requires security to be enabled and a platinum or better license
to run in FIPS mode.
Since not all FTR configs assume these conditions will be enabled, we
cant run every test. So these failing tests will be skipped when these
overrides are enforced.
This does not mean that the functionality is not supported in FIPS mode.
## What is the point?
Running these tests in FIPS mode is not necessarily to check that the
functionality works as expected, it is to make sure Kibana does not
crash due to unsupported algorithm usage (`md4`, `md5`, etc).
When running in FIPS mode, Node will throw an `unsupported envelope
function` error (with FIPS enabled) if it encounters an unsupported
algorithm, so the more lines of code covered, the more assurance we can
have that features will work in FIPS mode.
## Nature of the changes
To skip a test, a `tag` is added: `this.tags('skipFIPS')`
`this.tags` is only available for `describe('description', function()
{...});`
There should not be any logical changes, just tests wrapped in an extra
block.
I tried to make the wording in the new `describe` block "flow" 😅 if you
prefer different wording in the new `describe` block - please add a
change!
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Nikita Indik <nikita.indik@elastic.co>
## UPDATE
It has been removed the execution of the playwright tests on buildkite,
the execution will be re-enabled as soon as we are ready and as
described below in the PR, there are still steps pending to be done.
## Motivation
**Cypress is not performing well lately.**
* We have been facing significant performance issues with Cypress. For
instance, it takes a long time to open the visual interface and start
executing tests.
**Teams are finding it increasingly challenging to write new tests and
debug existing ones.**
* The time and effort required to create new tests or troubleshoot
existing ones have become burdensome.
**Concern about the impact this could have on our testing practices.**
* Lose motivation to write tests or, worse, skip writing crucial tests.
## Why Playwright?
* Compared to Cypress, Playwright seems to be known for its faster
execution times and lower resource consumption. What could have a
positive impact by having faster feedback during development and
execution of new tests as well as more efficient use of CI resources.
* Provides powerful debugging tools which can make easier to write,
debug and execute tests.
* Seems to provide the same capabilities we currently use in our Cypress
tests.
* Given Playwright's active development and backing by Microsoft, it is
likely to continue evolving rapidly, making it a safe long-term choice.
Considering all the above, Playwright seems to be a strong candidate to
replace Cypress and address all the issues we are facing lately
regarding UI test automation.
## Objective of this POC
To write in Playwright a couple of tests we currently have on Cypress to
check the performance of the tool as well as the development experience.
The tests selected have been:
-
[enable_risk_score_redirect.cy.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_cypress/cypress/e2e/entity_analytics/dashboards/enable_risk_score_redirect.cy.ts)
- Owned by Entity Analytics team and selected by its simplicity since it
does not need any special setup to be executed and is short.
-
[manual_rule_run.cy.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/rule_gaps/manual_rule_run.cy.ts)
- Owned by Detection Engine team and selected because is short and adds
a bit more of complexity due to it needs of clean-up and setting up
initial data through the API.
## How to execute the tests
### Visual mode
- Navigate to: `x-pack/test/security_solution_playwright`
- Execute: `yarn open:ess` for ESS environment or `yarn open:serverless`
for serverless environment.
### Headless mode
- Navigate to: `x-pack/test/security_solution_playwright`
- Execute: `yarn run:ess` for ESS environment or `yarn run:serverless`
for serverless environment.
### From VScode
- Install `Playwright Test for VScode` extension by Microsoft
- Navigate to: `x-pack/test/security_solution_playwright`
- Execute: `yarn open:ess` for ESS environment or `yarn open:serverless`
for serverless environment.
- Open your IDE
- Click on the `Testing` icon
- On the `Test Explorer` click on the three dots to select the profile
you are going to execute `ess` or `serverless`
- Click on the test you want to execute or navigate to the spec file of
the test and execute it from the same spec.
## My experience
- Tests are way easier to implement than with Cypress.
- Playwright does not rely on chainable commands. Chainable commands on
Cypress can lead to confusing code.
- Without chainable commands, the flow of the tests is more explicit and
easier to understand.
- You can notice that the tool has been designed with Typescript in
mind.
- Is super easy to implement the Page Object Model pattern (POM).
- With POM the test code is clean and focused on "what" rather than
"how".
- Love the fact that you can execute the tests from the same IDE without
having to switch windows during test development.
- The visual mode execution gives you lots of information out of the
box.
## The scope of this PR
- Sets the initial infrastructure to write and execute tests with
Playwright.
- Has examples and set a basis about how to write tests using the POM.
- Allows the execution of the tests in ESS and serverless (just
stateless environment).
- Integrates the execution of the tests with buildkite.
## Pending to be done/investigate
- Proper readme
- How to split tests and PO between the different teams
- Good reports on CI
- Upload screenshots on CI
- Flaky test suite runner
- Complete the labeling
- Execution of the tests on MKI environments
## FAQ
**Can I start adding tests to playwright?**
Currently, you can explore and experiment with Playwright, but there is
still work pending to be done to make the tool officially usable.
**Why security engineering productivity is the owner of the playwright
folder?**
This is something temporary to make sure that good practices are
followed.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: dkirchan <diamantis.kirchantzoglou@elastic.co>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>
Co-authored-by: Jon <jon@budzenski.me>
## Summary
Follow-up to #190690
Most of API integration tests does not match the path pattern set in the
original PR (thanks @pheyos for catching it) and where not updated.
This PR updates `.eslintrc.js` with explicit patterns to lint
api_integration tests. Hopefully it is final change, but I rely on code
owners to double check it.
Most of the changes are trivial adjustments:
- duplicated before/after hooks `mocha/no-sibling-hooks`
- duplicated test titles `mocha/no-identical-title`
- async function in describe() `mocha/no-async-describe`
---------
Co-authored-by: Ash <1849116+ashokaditya@users.noreply.github.com>
## Summary
This PR enforces ESLint rules in FTR tests, in particular:
- `no-floating-promises` rule to catch unawaited Promises in
tests/services/page objects
_Why is it important?_
- Keep correct test execution order: cleanup code may run before the
async operation is completed, leading to unexpected behavior in
subsequent tests
- Accurate test results: If a test completes before an async operation
(e.g., API request) has finished, Mocha might report the test as passed
or failed based on incomplete context.
```
198:11 error Promises must be awaited, end with a call to .catch, end with a call to .then
with a rejection handler or be explicitly marked as ignored with the `void` operator
@typescript-eslint/no-floating-promises
```
<img width="716" alt="Screenshot 2024-08-20 at 14 04 43"
src="https://github.com/user-attachments/assets/9afffe4c-4b51-4790-964c-c44a76baed1e">
- recommended rules from
[eslint-mocha-plugin](https://www.npmjs.com/package/eslint-plugin-mocha)
including:
-
[no-async-describe](https://github.com/lo1tuma/eslint-plugin-mocha/blob/main/docs/rules/no-async-describe.md)
-
[no-identical-title.md](https://github.com/lo1tuma/eslint-plugin-mocha/blob/main/docs/rules/no-identical-title.md)
-
[no-sibling-hooks.md](https://github.com/lo1tuma/eslint-plugin-mocha/blob/main/docs/rules/no-sibling-hooks.md)
and others
Note for reviewers: some tests were skipped due to failures after
missing `await` was added. Most likely is a "false positive" case when
test is finished before async operation is actually completed. Please
work on fixing and re-enabling it
---------
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## [Security Solution] [Attack discovery] Attack Discovery RBAC / Display an upgrade CTA for the serverless essentials product tier
### Summary
This PR adds Role Based Access Control (RBAC) to Attack discovery.
Security users may enable or disable the new `Attack Discovery` RBAC feature shown in the figure below:
_Above: The new `Attack discovery` RBAC feature_
It is possible to for example, configure a custom role that enables Attack discovery, but disables the assistant, as illustrated by the table below:
| Role | License | Navigation visible | Show upsell | Upsell has actions | View in assistant enabled |
|-------------------------------------------|------------|--------------------|-------------|--------------------|---------------------------|
| `has_attack_discovery_all_assistant_none` | Basic | ✅ | ✅ | ✅ | ❌ |
| `has_attack_discovery_all_assistant_none` | Trial | ✅ | ❌ | ❌ | ❌ |
| `has_attack_discovery_all_assistant_none` | Platinum | ✅ | ✅ | ✅ | ❌ |
| `has_attack_discovery_all_assistant_none` | Enterprise | ✅ | ❌ | ❌ | ❌ |
_Above: An example role that enables Attack discovery, but disables the assistant_
See the `Desk Testing` section of this PR for details.
This PR also fixes an issue where Attack discovery does not display an upgrade call to action (CTA) for the serverless _essentials_ product tier, per the before and after screenshots below:
#### Before
_Above: Before the fix, an upgrade CTA is NOT displayed for the serverless essentials product tier_
#### After
_Above: After the fix, an upgrade CTA is displayed for the serverless essentials product tier_
The fix above is implemented by adopting the upselling framework.
### New Feature ID
This PR adds a new Feature ID for attack discovery:
```typescript
export const ATTACK_DISCOVERY_FEATURE_ID = 'securitySolutionAttackDiscovery' as const;
```
in `x-pack/packages/security-solution/features/src/constants.ts`
### Upselling framework usage
This PR updates the Attack discovery page to use the upselling framework via the following summarized steps:
1. Removed the branching logic from `x-pack/plugins/security_solution/public/attack_discovery/pages/upgrade/index.tsx`, and moved the component to an upselling `section` component in `x-pack/packages/security-solution/upselling/sections/attack_discovery/index.tsx`, where the component was renamed to `AttackDiscoveryUpsellingSection`.
This `section` component handles (just) the styling of the upselling message and actions (by itself, without the page wrapper), and receives the following props:
```typescript
interface Props {
actions?: React.ReactNode;
availabilityMessage: string;
upgradeMessage: string;
}
```
The self managed and serverless-specific actions and `i18n` messages are passed down via the components described in the later steps below.
2. Removed all previous references to the `Upgrade` component (and references to `useProductTypes`) from the Attack discovery page in `x-pack/plugins/security_solution/public/attack_discovery/pages/index.tsx`, because the framework manages the upgrade case instead of the page itself.
3. Created an upselling `page` component `AttackDiscoveryUpsellingPage` in `x-pack/packages/security-solution/upselling/pages/attack_discovery/index.tsx`.
This component handles (just) the styling of the _page_ that wraps the Attack discovery `section`. It passes the same props to the previously described `AttackDiscoveryUpsellingSection` component.
4. Created a self-managed-specific `AttackDiscoveryUpsellingPageESS` component in `x-pack/plugins/security_solution_ess/public/upselling/pages/attack_discovery/index.tsx`
This component passes self-managed-specific upgrade action buttons / links and `i18n` strings to the previously described `AttackDiscoveryUpsellingPage`
5. Also for self managed, added a new `AttackDiscoveryUpsellingPageLazy` component to the existing file: `x-pack/plugins/security_solution_ess/public/upselling/lazy_upselling.tsx`
This component lazy loads the previously described `AttackDiscoveryUpsellingPageESS` component.
6. Added registration for the previously described `AttackDiscoveryUpsellingPageLazy` component to the existing `UpsellingPages` section in `x-pack/plugins/security_solution_ess/public/upselling/register_upsellings.tsx` with a `minimumLicenseRequired` of `enterprise`:
```
minimumLicenseRequired: 'enterprise',
```
7. Created a serverless-specific `AttackDiscoveryUpsellingPageServerless` component in `x-pack/plugins/security_solution_serverless/public/upselling/pages/attack_discovery/index.tsx`
This component passes serverless-specific `i18n` messages to the platform agnostic `AttackDiscoveryUpsellingPage` component.
8. Also for serverless, added a new `AttackDiscoveryUpsellingPageLazy` component to the existing file: `x-pack/plugins/security_solution_serverless/public/upselling/lazy_upselling.tsx`
9. Added registration for the previously described `AttackDiscoveryUpsellingPageLazy` component to the existing `upsellingPages` section in `x-pack/plugins/security_solution_serverless/public/upselling/upsellings.tsx` with the `assistant` PLI:
```
pli: ProductFeatureKey.assistant,
```
10. Added the `${ASSISTANT_FEATURE_ID}.ai-assistant` capability as an OR condition (via nested array, per the [framework](https://github.com/elastic/kibana/blob/main/x-pack/plugins/security_solution/public/common/lib/capabilities/has_capabilities.ts#L11-L22)) to the Attack discovery link in `x-pack/plugins/security_solution/public/attack_discovery/links.ts`. This addition enables the security route wrapper to check for upselling pages in serverless:
```
capabilities: [[`${SERVER_APP_ID}.show`, `${ASSISTANT_FEATURE_ID}.ai-assistant`]],
```
11. Added `licenseType: 'enterprise'` to the Attack discovery link in `x-pack/plugins/security_solution/public/attack_discovery/links.ts` to require an `enterprise` license for self managed
### Upgrade CTA gallery
The screenshots in this section document the CTA (or Welcome message when the feature is licensed) displayed for various license levels after the fix:
#### Users with the `None` privilege
If users with the `None` privilege manually enter an Attack discovery URL, e.g. `http://localhost:5601/app/security/attack_discovery`, the framework will display the following error prompt:
#### Self managed BASIC
#### Self managed PLATINUM
#### Self managed TRIAL
#### Self managed ENTERPRISE
#### Serverless ESSENTIALS
#### Serverless COMPLETE
## Desk Testing
### Severless: Desk testing (just) the upgrade CTA
Reproduction steps:
1) Comment-out any preconfigured connectors in `config/kibana.dev.yml`
2) Edit the configuration of `config/serverless.security.yml` to enable the `essentials` product tier:
```yaml
xpack.securitySolutionServerless.productTypes:
[
{ product_line: 'security', product_tier: 'essentials' }
]
```
3) Start Elasticsearch
```sh
yarn es serverless --projectType security
```
4) Start a development instance of Kibana
```
yarn start --serverless=security --no-base-path
```
5) Navigate to Security > Attack discovery
**Expected result**
- An upgrade CTA is displayed, as illustrated by the _after_ screenshot below:
- The video tour is NOT displayed for serverless, as noted in the [original PR](https://github.com/elastic/kibana/pull/182605#issuecomment-2100607857)
**Actual result**
- An upgrade CTA is NOT displayed, as illustrated by the _before_ screenshot below:
### Desk testing Self Managed
To desk test self manged, we will:
1) Create (three) roles for testing
2) Create (three) users assigned to the roles
3) Test each role at `Basic`, `Trial`, `Platinum`, and `Enterprise` levels to verify:
- `Attack discovery` link visibility in the Security solution navigation
- Visibility of the upsell empty prompt for license levels where Attack discovery is unavailable
- The upsell empty prompt includes the `Subscription plans` and `Manage license` actions
- When Attack discoveries are generated, the `View in Ai Assistant` button and popover menu action are enabled / disabled, based on availability of the `AI Assistant` feature
#### Creating (three) roles for testing
In this section, we will start a new (development) self managed deployment, and create the following three roles via Kibana Dev Tools:
- `has_attack_discovery_all_assistant_all`
- `has_attack_discovery_all_assistant_none`
- `has_attack_discovery_none_assistant_all`
To start the deployment and create the roles:
1) Add a pre-configured GenAI connector to `config/kibana.dev.yml`
2) Start a new (development) instance of Elasticsearch:
```sh
yarn es snapshot -E path.data=/Users/$USERNAME/data-2024-07-31a
```
3) Start a local (development) instance of Kibana:
```
yarn start --no-base-path
````
4) Login to Kibana as the `elastic` user
5) Generate some alerts
6) Navigate to Dev Tools
7) Execute the following three API calls to create the roles:
<details><summary>PUT /_security/role/has_attack_discovery_all_assistant_all</summary>
<p>
``` ts
PUT /_security/role/has_attack_discovery_all_assistant_all
{
"cluster": [
"all"
],
"indices": [
{
"names": [
"*"
],
"privileges": [
"all"
],
"field_security": {
"grant": [
"*"
],
"except": []
},
"allow_restricted_indices": false
}
],
"applications": [
{
"application": "kibana-.kibana",
"privileges": [
"feature_securitySolutionAssistant.minimal_all",
"feature_securitySolutionAttackDiscovery.minimal_all",
"feature_siem.all",
"feature_securitySolutionCases.all",
"feature_actions.all"
],
"resources": [
"*"
]
}
],
"run_as": [],
"metadata": {},
"transient_metadata": {
"enabled": true
}
}
```
</p>
</details>
<details><summary>PUT /_security/role/has_attack_discovery_all_assistant_none</summary>
<p>
``` ts
PUT /_security/role/has_attack_discovery_all_assistant_none
{
"cluster": [
"all"
],
"indices": [
{
"names": [
"*"
],
"privileges": [
"all"
],
"field_security": {
"grant": [
"*"
],
"except": []
},
"allow_restricted_indices": false
}
],
"applications": [
{
"application": "kibana-.kibana",
"privileges": [
"feature_securitySolutionAttackDiscovery.minimal_all",
"feature_siem.all",
"feature_securitySolutionCases.all",
"feature_actions.all"
],
"resources": [
"*"
]
}
],
"run_as": [],
"metadata": {},
"transient_metadata": {
"enabled": true
}
}
```
</p>
</details>
<details><summary>PUT /_security/role/has_attack_discovery_none_assistant_all</summary>
<p>
``` ts
PUT /_security/role/has_attack_discovery_none_assistant_all
{
"cluster": [
"all"
],
"indices": [
{
"names": [
"*"
],
"privileges": [
"all"
],
"field_security": {
"grant": [
"*"
],
"except": []
},
"allow_restricted_indices": false
}
],
"applications": [
{
"application": "kibana-.kibana",
"privileges": [
"feature_securitySolutionAssistant.minimal_all",
"feature_siem.all",
"feature_securitySolutionCases.all",
"feature_actions.all"
],
"resources": [
"*"
]
}
],
"run_as": [],
"metadata": {},
"transient_metadata": {
"enabled": true
}
}
```
</p>
</details>
#### Creating (three) users assigned to the roles
In this section, we will create the following three users via Kibana Dev Tools using the API calls below (expand for details):
- `attack_discovery_all_assistant_all`
- `attack_discovery_all_assistant_none`
- `attack_discovery_none_assistant_all`
1) Navigate to Dev Tools
2) Execute the following three API calls to create the users:
<details><summary>POST /_security/user/attack_discovery_all_assistant_all</summary>
<p>
``` ts
POST /_security/user/attack_discovery_all_assistant_all
{
"username": "attack_discovery_all_assistant_all",
"password": "changeme",
"roles": [
"has_attack_discovery_all_assistant_all"
],
"full_name": "Attack Discovery All Assistant All",
"email": "user@example.com",
"metadata": {},
"enabled": true
}
```
</p>
</details>
<details><summary>POST /_security/user/attack_discovery_all_assistant_none</summary>
<p>
``` ts
POST /_security/user/attack_discovery_all_assistant_none
{
"username": "attack_discovery_all_assistant_none",
"password": "changeme",
"roles": [
"has_attack_discovery_all_assistant_none"
],
"full_name": "Attack Discovery All Assistant None",
"email": "user@example.com",
"metadata": {},
"enabled": true
}
```
</p>
</details>
<details><summary>POST /_security/user/attack_discovery_none_assistant_all</summary>
<p>
``` ts
POST /_security/user/attack_discovery_none_assistant_all
{
"username": "attack_discovery_none_assistant_all",
"password": "changeme",
"roles": [
"has_attack_discovery_none_assistant_all"
],
"full_name": "Attack Discovery None Assistant All",
"email": "user@example.com",
"metadata": {},
"enabled": true
}
```
</p>
</details>
#### Testing each role at `Basic`, `Trial`, `Platinum`, and `Enterprise` levels
In this section, we will test each of the self managed `Basic`, `Trial`, `Platinum`, and `Enterprise` license levels with the three roles we created for testing.
##### Testing the `has_attack_discovery_all_assistant_all` role
1) Login as the `attack_discovery_all_assistant_all` user
2) Navigate to the Security solution
3) For each of the `Basic`, `Trial`, `Platinum`, and `Enterprise` levels, verify your observations match the expected behavior in the table below:
| Role | License | Navigation visible | Show upsell | Upsell has actions | View in assistant enabled |
|------------------------------------------|------------|--------------------|-------------|--------------------|---------------------------|
| `has_attack_discovery_all_assistant_all` | Basic | ✅ | ✅ | ✅ | ❌ |
| `has_attack_discovery_all_assistant_all` | Trial | ✅ | ❌ | ❌ | ✅ |
| `has_attack_discovery_all_assistant_all` | Platinum | ✅ | ✅ | ✅ | ❌ |
| `has_attack_discovery_all_assistant_all` | Enterprise | ✅ | ❌ | ❌ | ✅ |
##### Testing the `has_attack_discovery_all_assistant_none` role
1) Login as the `attack_discovery_all_assistant_none` user
2) Navigate to the Security solution
3) For each of the `Basic`, `Trial`, `Platinum`, and `Enterprise` levels, verify your observations match the expected behavior in the table below:
| Role | License | Navigation visible | Show upsell | Upsell has actions | View in assistant enabled |
|-------------------------------------------|------------|--------------------|-------------|--------------------|---------------------------|
| `has_attack_discovery_all_assistant_none` | Basic | ✅ | ✅ | ✅ | ❌ |
| `has_attack_discovery_all_assistant_none` | Trial | ✅ | ❌ | ❌ | ❌ |
| `has_attack_discovery_all_assistant_none` | Platinum | ✅ | ✅ | ✅ | ❌ |
| `has_attack_discovery_all_assistant_none` | Enterprise | ✅ | ❌ | ❌ | ❌ |
##### Testing the `has_attack_discovery_none_assistant_all` role
1) Login as the `attack_discovery_none_assistant_all` user
2) Navigate to the Security solution
3) For each of the `Basic`, `Trial`, `Platinum`, and `Enterprise` levels, verify your observations match the expected behavior in the table below:
| Role | License | Navigation visible | Show upsell | Upsell has actions | View in assistant enabled |
|-------------------------------------------|------------|--------------------|-------------|--------------------|---------------------------|
| `has_attack_discovery_none_assistant_all` | Basic | ✅ | ✅ | ✅ | ❌ |
| `has_attack_discovery_none_assistant_all` | Trial | ❌ | ❌ | ❌ | ❌ |
| `has_attack_discovery_none_assistant_all` | Platinum | ✅ | ✅ | ✅ | ❌ |
| `has_attack_discovery_none_assistant_all` | Enterprise | ❌ | ❌ | ❌ | ❌ |
---------------------------------------------
### Serverless Testing
To desk test serverless, we will test the `essentials` and `complete` product tiers to verify:
- `Attack discovery` link visibility in the Security project navigation
- Visibility of the upsell empty prompt for license levels where Attack discovery is unavailable
- The upsell empty prompt does NOT include the `Subscription plans` and `Manage license` actions
- When Attack discoveries are generated, the `View in Ai Assistant` button and popover menu action are enabled
#### Essentials tier testing
1) Add a pre-configured GenAI connector to `config/kibana.dev.yml`
2) Edit the configuration of `config/serverless.security.yml` to enable the `essentials` product tier:
```yaml
xpack.securitySolutionServerless.productTypes:
[
{ product_line: 'security', product_tier: 'essentials' },
{ product_line: 'endpoint', product_tier: 'essentials' },
]
```
2) Start a new (development) instance of Elasticsearch:
```sh
yarn es serverless --clean --projectType security
```
3) Start a local (development) instance of Kibana:
```
yarn start --serverless=security --no-base-path
````
4) select the `admin` role
5) Generate some alerts
6) Verify your observations match the expected behavior in the table below:
| Role | Tier | Navigation visible | Show upsell | Upsell has actions | View in assistant enabled |
|-------------------------------|------------|--------------------|-------------|--------------------|---------------------------|
| `viewer` | essentials | ✅ | ✅ | ❌ | ❌ |
| `editor` | essentials | ✅ | ✅ | ❌ | ❌ |
| `t1_analyst` | essentials | ✅ | ✅ | ❌ | ❌ |
| `t2_analyst` | essentials | ✅ | ✅ | ❌ | ❌ |
| `t3_analyst` | essentials | ✅ | ✅ | ❌ | ❌ |
| `threat_intelligence_analyst` | essentials | ✅ | ✅ | ❌ | ❌ |
| `rule_author` | essentials | ✅ | ✅ | ❌ | ❌ |
| `soc_manager` | essentials | ✅ | ✅ | ❌ | ❌ |
| `detections_admin` | essentials | ✅ | ✅ | ❌ | ❌ |
| `platform_engineer` | essentials | ✅ | ✅ | ❌ | ❌ |
| `endpoint_operations_analyst` | essentials | ✅ | ✅ | ❌ | ❌ |
| `endpoint_policy_manager` | essentials | ✅ | ✅ | ❌ | ❌ |
| `admin` | essentials | ✅ | ✅ | ❌ | ❌ |
| `system_indices_superuser` | essentials | ✅ | ✅ | ❌ | ❌ |
### Complete tier testing
1) Stop the running Kibana server (from the previous Essentials tier testing)
2) Edit the configuration of `config/serverless.security.yml` to enable the `complete` product tier:
```yaml
xpack.securitySolutionServerless.productTypes:
[
{ product_line: 'security', product_tier: 'complete' },
{ product_line: 'endpoint', product_tier: 'complete' },
]
```
3) Restart a local (development) instance of Kibana:
```
yarn start --serverless=security --no-base-path
````
4) Verify your observations match the expected behavior in the table below:
| Role | Tier | Navigation visible | Show upsell | Upsell has actions | View in assistant enabled |
|-------------------------------|----------|--------------------|-------------|--------------------|---------------------------|
| `viewer` | complete | ✅ | ❌ | ❌ | ✅ |
| `editor` | complete | ✅ | ❌ | ❌ | ✅ |
| `t1_analyst` | complete | ✅ | ❌ | ❌ | ✅ |
| `t2_analyst` | complete | ✅ | ❌ | ❌ | ✅ |
| `t3_analyst` | complete | ✅ | ❌ | ❌ | ✅ |
| `threat_intelligence_analyst` | complete | ✅ | ❌ | ❌ | ✅ |
| `rule_author` | complete | ✅ | ❌ | ❌ | ✅ |
| `soc_manager` | complete | ✅ | ❌ | ❌ | ✅ |
| `detections_admin` | complete | ✅ | ❌ | ❌ | ✅ |
| `platform_engineer` | complete | ✅ | ❌ | ❌ | ✅ |
| `endpoint_operations_analyst` | complete | ✅ | ❌ | ❌ | ✅ |
| `endpoint_policy_manager` | complete | ✅ | ❌ | ❌ | ✅ |
| `admin` | complete | ✅ | ❌ | ❌ | ✅ |
| `system_indices_superuser` | complete | ✅ | ❌ | ❌ | ✅ |
## Summary
While working on #188737 I had to move `supertestWithoutAuth` into
`kbn-ftr-common-functional-services` package. This change seems to be
bigger than initially planned.
Moving it to the separate PR with following changes:
- move FTR `SupertestWithoutAuthProvider` service to package
- remove "duplicates" in favour of service from package
- update service type where needed
## Summary
At the moment, our package generator creates all packages with the type
`shared-common`. This means that we cannot enforce boundaries between
server-side-only code and the browser, and vice-versa.
- [x] I started fixing `packages/core/*`
- [x] It took me to fixing `src/core/` type to be identified by the
`plugin` pattern (`public` and `server` directories) vs. a package
(either common, or single-scoped)
- [x] Unsurprisingly, this extended to packages importing core packages
hitting the boundaries eslint rules. And other packages importing the
latter.
- [x] Also a bunch of `common` logic that shouldn't be so _common_ 🙃
### For maintainers
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Passing default solution from cloud onboarding process.
1. Renaming. Solution changes are not released yet, would be shipped
with `8.15`, so it's fine to do it.
- `search` -> `es`
- `observability` -> `oblt`
- Adjusted telemetry accordingly
2. Added `cloud` as optional dependency to `spaces` plugin to use
`onboarding.defaultSolution` passed through setup contract.
### How to test
1. Set `xpack.cloud.onboarding.default_solution` to `es | oblt |
security`
2. Check that default space was created with provided solution `GET
kbn:/api/spaces/space/default`
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### For maintainers
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
__Fixes: https://github.com/elastic/kibana/issues/184999__
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Added telemetry for space solution property.
### How to test
```
# Create a couple of spaces with solution
POST kbn:/api/spaces/space
{
"name": "space with solution",
"id": "my-space-solution-1",
"description": "a description",
"color": "#5c5959",
"solution": "search",
"disabledFeatures": []
}
POST kbn:/api/spaces/space
{
"name": "space with solution",
"id": "my-space-solution-2",
"description": "a description",
"color": "#5c5959",
"solution": "search",
"disabledFeatures": []
}
POST kbn:/api/spaces/space
{
"name": "space with solution",
"id": "my-space-solution-3",
"description": "a description",
"color": "#5c5959",
"solution": "security",
"disabledFeatures": []
}
# Get stats data
POST kbn:/internal/telemetry/clusters/_stats
{
"unencrypted": true, "refreshCache": true
}
```
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### For maintainers
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
__Fixes: https://github.com/elastic/kibana/issues/183641__
## Release Note
Added telemetry for space solution property.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
We aggregate on disabledFeatures in [Spaces Usage
Collector](5e95a76796/x-pack/plugins/spaces/server/usage_collection/spaces_usage_collector.ts (L38)),
but field was removed from mappings. Added `disabledFeatures` back to
mappings.
### How to Test
1. Create a couple of spaces with disabled features.
```
POST kbn:/api/spaces/space
{
"name": "my-space-1",
"id": "my-space-1",
"description": "a description",
"color": "#5c5959",
"disabledFeatures": ["canvas", "discover"]
}
POST kbn:/api/spaces/space
{
"name": "my-space-2",
"id": "my-space-2",
"description": "a description",
"color": "#5c5959",
"disabledFeatures": ["savedObjectsManagement", "canvas"]
}
```
2. Make a request to stats endpoint and check that `disabledFeatures`
counters.
```
POST kbn:/internal/telemetry/clusters/_stats
{
"unencrypted": true, "refreshCache": true
}
```
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### For maintainers
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
__Fixes: https://github.com/elastic/kibana/issues/184194__
## Release note
Added `disabledFeatures` back to mappings, so it can be aggregated on.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Added solution property for the space.
- Forbidden in serverless.
- To facilitate iterative development made the property as optional in
stateful offering until all of the workstreams are complete.
### How to test API changes
```
# Should create space
POST kbn:/api/spaces/space
{
"name": "space without solution",
"id": "my-space-solution-1",
"description": "a description",
"color": "#5c5959",
"disabledFeatures": []
}
# Should fail with 400
POST kbn:/api/spaces/space
{
"name": "space with solution",
"id": "my-space-solution-2",
"description": "a description",
"color": "#5c5959",
"solution": "some_solution",
"disabledFeatures": []
}
# Should fail with 400
POST kbn:/api/spaces/space
{
"name": "space with solution",
"id": "my-space-solution-2",
"description": "a description",
"color": "#5c5959",
"solution": null,
"disabledFeatures": []
}
# Should create space
POST kbn:/api/spaces/space
{
"name": "space with solution",
"id": "my-space-solution-2",
"description": "a description",
"color": "#5c5959",
"solution": "search",
"disabledFeatures": []
}
# Should get 'my-space-solution-1' space without solution field
GET kbn:/api/spaces/space/my-space-solution-1
# Should get 'my-space-solution-2' space with solution field
GET kbn:/api/spaces/space/my-space-solution-2
# Should fail to update with 400
PUT kbn:/api/spaces/space/my-space-solution-1
{
"id": "my-space-solution-1",
"name": "my-space-solution-1 name",
"solution": "some_solution"
}
# Should fail to update with 400
PUT kbn:/api/spaces/space/my-space-solution-1
{
"id": "my-space-solution-1",
"name": "my-space-solution-1 name",
"solution": null
}
# Should update 'my-space-solution-1'
PUT kbn:/api/spaces/space/my-space-solution-1
{
"id": "my-space-solution-1",
"name": "my-space-solution-1 name",
"solution": "security"
}
# Should get 'my-space-solution-1' space wit solution field set to 'security'
GET kbn:/api/spaces/space/my-space-solution-1
# Should return list where
# 1. 'my-space-solution-1' has solution 'security'
# 2. 'my-space-solution-2' has solution 'search'
# 3. Other spaces don't have solution field present
GET kbn:/api/spaces/space
```
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed ([Security and Spaces
config](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6076),
[Spaces only
config](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6075))
### For maintainers
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
__Fixes: https://github.com/elastic/kibana/issues/183559__
## Release note
Added optional solution property for Space in a stateful offering.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Related to https://github.com/elastic/kibana/issues/7104
Update supertest, superagent, and the corresponding type package, to
their latest version.
(of course, types had some signature changes and we're massively using
supertest in all our FTR suites so the whole Kibana multiverse has to
review it)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Closes#168049
## Summary
This PR adjusts the KQL filters when collecting references to saved
objects for the purpose of updating their spaces (i.e. sharing saved
objects to spaces). An additional filter is added to specifically
exclude ID -> ID matches - an ID match would mean the object has already
been shared to the destination space and there is no conflict. Filters
to match the shared object's ID or origin ID to the destination space's
objects' origin ID, and to match the shared object's origin ID to the
destination space's objects' IDs remain in place to properly check for
conflicts with potential copies.
### Manual Testing
- Create 2 spaces: A and B
- Add a sample data set (e.g. flight) to space A
- In Discover, create a saved query called "s1" with a filter pill that
uses the sample data logs data view
- Create another saved query called "s2" with a filter pill that uses
the sample data logs data view
- Go to `Stack Management->Saved` Objects and share the "s1" query to
space B
- Now share the "s2" query to space B. From the main branch you will see
that there is a conflict that disallows sharing the second query. This
is because it is also attempting to share the referenced data view,
which is already in space B. However, this should not be a conflict -
from this PR you will be able to successfully share both queries.
### Automated Testing
-
packages/core/saved-objects/core-saved-objects-api-server-internal/src/lib/apis/utils/find_shared_origin_objects.test.ts
-
packages/core/saved-objects/core-saved-objects-api-server-internal/src/lib/apis/internals/collect_multi_namespace_references.test.ts
-
x-pack/test/spaces_api_integration/common/suites/get_shareable_references.ts
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Closes
#155723,#155846,#156045,#156041,#156074,#156130,#156151,#156739,#157673,#156422,#156998,#156922,#156921,#157452,#158366,#158586,#158711
## Summary
Issue was resolved by PR #160584. This PR unskips saved object tests
documented as skipped in #158918.
### Testing
Flaky test runner:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2543
## Summary
Addresses root cause of https://github.com/elastic/kibana/issues/158918
Underlying cause is that _esArchiver_ is messing up with the SO indices
whilst Kibana is already running.
This can cause some asynchronous calls made by Kibana (e.g. `GET
/.kibana_8.8.0/telemetry:telemetry`) to hit ES at the exact time where
the underlying SO indices are **just** recreated, causing the error
described in the related issue.
The idea of the fix is to delete `mappings.json`, used by _esArchiver_
to create the SO indices. This way, _esArchiver_ will use existing SO
indices instead (aka the "official" ones, created by Kibana at startup),
thus avoiding the problem altogether.
As a side effect:
- Documents in `data.json` must be updated so that they are correctly
inserted.
- The different FTR tests must make sure the SO indices are empty before
inserting those documents (done in the `before(), beforeEach()`
statements).
## Summary
Skips several tests failing with the
`no_shard_available_action_exception`. Also adds comments noting the
related test failure issues and the [master issue being
tracked](https://github.com/elastic/kibana/issues/158918).
See issue #158918 and PR #159002.
## Summary
Splitting ftr configs to speed up ftr tests run on CI
```
The following "Functional Tests" configs have durations that exceed the maximum amount of time desired for a single CI job.
This is not an error, and if you don't own any of these configs then you can ignore this warning.If you own any of these
configs please split them up ASAP and ask Operations if you have questions about how to do that.
x-pack/test/spaces_api_integration/security_and_spaces/config_basic.ts: 38.2 minutes
x-pack/test/spaces_api_integration/security_and_spaces/config_trial.ts: 38.2 minutes
```
After split:
x-pack/test/spaces_api_integration/security_and_spaces/config_basic.ts
20m 29s
x-pack/test/spaces_api_integration/security_and_spaces/copy_to_space_config_basic.ts
20m 52s
x-pack/test/spaces_api_integration/security_and_spaces/config_trial.ts
19m 57s
x-pack/test/spaces_api_integration/security_and_spaces/copy_to_space_config_trial.ts
21m 5s
Rebalancing it with other configs should speedup CI by at least few
minutes
## Description
Fix https://github.com/elastic/kibana/issues/104081
This PR move some of the SO types from the `.kibana` index into the
following ones:
- `.kibana_alerting_cases`
- `.kibana_analytics`
- `.kibana_security_solution`
- `.kibana_ingest`
This split/reallocation will occur during the `8.8.0` Kibana upgrade
(*meaning: from any version older than `8.8.0` to any version greater or
equal to `8.8.0`*)
**This PR main changes are:**
- implement the changes required in the SO migration algorithm to
support this reallocation
- update the FTR tools (looking at you esArchiver) to support these new
indices
- update hardcoded references to `.kibana` and usage of the
`core.savedObjects.getKibanaIndex()` to use new APIs to target the
correct index/indices
- update FTR datasets, tests and utility accordingly
## To reviewers
**Overall estimated risk of regressions: low**
But, still, please take the time to review changes in your code. The
parts of the production code that were the most impacted are the
telemetry collectors, as most of them were performing direct requests
against the `.kibana` index, so we had to adapt them. Most other
contributor-owned changes are in FTR tests and datasets.
If you think a type is misplaced (either we missed some types that
should be moved to a specific index, or some types were moved and
shouldn't have been) please tell us, and we'll fix the reallocation
either in this PR or in a follow-up.
## .Kibana split
The following new indices are introduced by this PR, with the following
SO types being moved to it. (any SO type not listed here will be staying
in its current index)
Note: The complete **_type => index_** breakdown is available in [this
spreadsheet](https://docs.google.com/spreadsheets/d/1b_MG_E_aBksZ4Vkd9cVayij1oBpdhvH4XC8NVlChiio/edit#gid=145920788).
#### `.kibana_alerting_cases`
- action
- action_task_params
- alert
- api_key_pending_invalidation
- cases
- cases-comments
- cases-configure
- cases-connector-mappings
- cases-telemetry
- cases-user-actions
- connector_token
- rules-settings
- maintenance-window
#### `.kibana_security_solution`
- csp-rule-template
- endpoint:user-artifact
- endpoint:user-artifact-manifest
- exception-list
- exception-list-agnostic
- osquery-manager-usage-metric
- osquery-pack
- osquery-pack-asset
- osquery-saved-query
- security-rule
- security-solution-signals-migration
- siem-detection-engine-rule-actions
- siem-ui-timeline
- siem-ui-timeline-note
- siem-ui-timeline-pinned-event
#### `.kibana_analytics`
- canvas-element
- canvas-workpad-template
- canvas-workpad
- dashboard
- graph-workspace
- index-pattern
- kql-telemetry
- lens
- lens-ui-telemetry
- map
- search
- search-session
- search-telemetry
- visualization
#### `.kibana_ingest`
- epm-packages
- epm-packages-assets
- fleet-fleet-server-host
- fleet-message-signing-keys
- fleet-preconfiguration-deletion-record
- fleet-proxy
- ingest_manager_settings
- ingest-agent-policies
- ingest-download-sources
- ingest-outputs
- ingest-package-policies
## Tasks / PRs
### Sub-PRs
**Implementation**
- 🟣https://github.com/elastic/kibana/pull/154846
- 🟣https://github.com/elastic/kibana/pull/154892
- 🟣https://github.com/elastic/kibana/pull/154882
- 🟣https://github.com/elastic/kibana/pull/154884
- 🟣https://github.com/elastic/kibana/pull/155155
**Individual index split**
- 🟣https://github.com/elastic/kibana/pull/154897
- 🟣https://github.com/elastic/kibana/pull/155129
- 🟣https://github.com/elastic/kibana/pull/155140
- 🟣https://github.com/elastic/kibana/pull/155130
### Improvements / follow-ups
- 👷🏼 Extract logic into
[runV2Migration](https://github.com/elastic/kibana/pull/154151#discussion_r1158470566)
@gsoldevila
- Make `getCurrentIndexTypesMap` resillient to intermittent failures
https://github.com/elastic/kibana/pull/154151#discussion_r1169289717
- 🚧 Build a more structured
[MigratorSynchronizer](https://github.com/elastic/kibana/pull/154151#discussion_r1158469918)
- 🟣https://github.com/elastic/kibana/pull/155035
- 🟣https://github.com/elastic/kibana/pull/155116
- 🟣https://github.com/elastic/kibana/pull/155366
## Reallocation tweaks
Tweaks to the reallocation can be done after the initial merge, as long
as it's done before the public release of 8.8
- `url` should get back to `.kibana` (see
[comment](https://github.com/elastic/kibana/pull/154888#discussion_r1172317133))
## Release Note
For performance purposes, Kibana is now using more system indices to
store its internal data.
The following system indices will be created when upgrading to `8.8.0`:
- `.kibana_alerting_cases`
- `.kibana_analytics`
- `.kibana_security_solution`
- `.kibana_ingest`
---------
Co-authored-by: pgayvallet <pierre.gayvallet@elastic.co>
Co-authored-by: Christos Nasikas <christos.nasikas@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>
Fixes https://github.com/elastic/kibana/issues/149344
This PR migrates all plugins to packages automatically. It does this
using `node scripts/lint_packages` to automatically migrate
`kibana.json` files to `kibana.jsonc` files. By doing this automatically
we can simplify many build and testing procedures to only support
packages, and not both "packages" and "synthetic packages" (basically
pointers to plugins).
The majority of changes are in operations related code, so we'll be
having operations review this before marking it ready for review. The
vast majority of the code owners are simply pinged because we deleted
all `kibana.json` files and replaced them with `kibana.jsonc` files, so
we plan on leaving the PR ready-for-review for about 24 hours before
merging (after feature freeze), assuming we don't have any blockers
(especially from @elastic/kibana-core since there are a few core
specific changes, though the majority were handled in #149370).
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
The location of plugins was previously somewhat irrelevant, but as we
move into packages it's more important that we can find all plugins in
the repository, and we would like to be able to do that without needing
to maintain a manifest somewhere to accomplish this. In order to make
this possible we plan to find any plugin/package by spotting all
kibana.json files which are not "fixtures". This allows plugin-like code
(but not actual plugin code) to exist for testing purposes, but it must
be within some form of "fixtures" directory, and any plugin that isn't
in a fixtures directory will be automatically pulled into the system
(though test plugins, examples, etc. will still only be loaded when the
plugin's path is passed via `--plugin-path`, the system will know about
them and use that knowledge for other things).
Since this is just a rename Operations will review and merge by EOD Jan
12th unless someone has a blocking concern.
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Dearest Reviewers 👋
I've been working on this branch with @mistic and @tylersmalley and
we're really confident in these changes. Additionally, this changes code
in nearly every package in the repo so we don't plan to wait for reviews
to get in before merging this. If you'd like to have a concern
addressed, please feel free to leave a review, but assuming that nobody
raises a blocker in the next 24 hours we plan to merge this EOD pacific
tomorrow, 12/22.
We'll be paying close attention to any issues this causes after merging
and work on getting those fixed ASAP. 🚀
---
The operations team is not confident that we'll have the time to achieve
what we originally set out to accomplish by moving to Bazel with the
time and resources we have available. We have also bought ourselves some
headroom with improvements to babel-register, optimizer caching, and
typescript project structure.
In order to make sure we deliver packages as quickly as possible (many
teams really want them), with a usable and familiar developer
experience, this PR removes Bazel for building packages in favor of
using the same JIT transpilation we use for plugins.
Additionally, packages now use `kbn_references` (again, just copying the
dx from plugins to packages).
Because of the complex relationships between packages/plugins and in
order to prepare ourselves for automatic dependency detection tools we
plan to use in the future, this PR also introduces a "TS Project Linter"
which will validate that every tsconfig.json file meets a few
requirements:
1. the chain of base config files extended by each config includes
`tsconfig.base.json` and not `tsconfig.json`
1. the `include` config is used, and not `files`
2. the `exclude` config includes `target/**/*`
3. the `outDir` compiler option is specified as `target/types`
1. none of these compiler options are specified: `declaration`,
`declarationMap`, `emitDeclarationOnly`, `skipLibCheck`, `target`,
`paths`
4. all references to other packages/plugins use their pkg id, ie:
```js
// valid
{
"kbn_references": ["@kbn/core"]
}
// not valid
{
"kbn_references": [{ "path": "../../../src/core/tsconfig.json" }]
}
```
5. only packages/plugins which are imported somewhere in the ts code are
listed in `kbn_references`
This linter is not only validating all of the tsconfig.json files, but
it also will fix these config files to deal with just about any
violation that can be produced. Just run `node scripts/ts_project_linter
--fix` locally to apply these fixes, or let CI take care of
automatically fixing things and pushing the changes to your PR.
> **Example:** [`64e93e5`
(#146212)](64e93e5806)
When I merged main into my PR it included a change which removed the
`@kbn/core-injected-metadata-browser` package. After resolving the
conflicts I missed a few tsconfig files which included references to the
now removed package. The TS Project Linter identified that these
references were removed from the code and pushed a change to the PR to
remove them from the tsconfig.json files.
## No bazel? Does that mean no packages??
Nope! We're still doing packages but we're pretty sure now that we won't
be using Bazel to accomplish the 'distributed caching' and 'change-based
tasks' portions of the packages project.
This PR actually makes packages much easier to work with and will be
followed up with the bundling benefits described by the original
packages RFC. Then we'll work on documentation and advocacy for using
packages for any and all new code.
We're pretty confident that implementing distributed caching and
change-based tasks will be necessary in the future, but because of
recent improvements in the repo we think we can live without them for
**at least** a year.
## Wait, there are still BUILD.bazel files in the repo
Yes, there are still three webpack bundles which are built by Bazel: the
`@kbn/ui-shared-deps-npm` DLL, `@kbn/ui-shared-deps-src` externals, and
the `@kbn/monaco` workers. These three webpack bundles are still created
during bootstrap and remotely cached using bazel. The next phase of this
project is to figure out how to get the package bundling features
described in the RFC with the current optimizer, and we expect these
bundles to go away then. Until then any package that is used in those
three bundles still needs to have a BUILD.bazel file so that they can be
referenced by the remaining webpack builds.
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Merges the changes of #134395 into the new packages structure.
Resolves#133835
### Description
This PR represents a fully manual merge of the saved objects refactor of
client wrapper system into repository extensions. These changes are
being manually merged due to significant changes of the saved objects
implementation in the main branch, specifically the migration to the new
packages structure.
### Other changes
- Bulk Delete: bulk delete was implemented in parallel to #134395 being
completed and this PR will refactor that API to utilize the new
extensions
Co-authored-by: Jeramy Soucy <jeramy.soucy@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>
* Removing esArchiver in favor of testDataLoader
* [CI] Auto-commit changed files from 'node scripts/precommit_hook.js --ref HEAD~1..HEAD --fix'
* Adding test data for loader
* [CI] Auto-commit changed files from 'node scripts/precommit_hook.js --ref HEAD~1..HEAD --fix'
* Adding generic TestDataLoader
* Importing just the type per PR feedback
* Changing testDataLoader function names to be more descriptive
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Larry Gregory <larry.gregory@elastic.co>
Co-authored-by: Thomas Watson <watson@elastic.co>
