Partially addresses https://github.com/elastic/kibana/issues/159590
## Summary
This PR disables the Advanced settings plugin for all projects in
serverless.
**How to test:**
1. Start Elasticsearch with `yarn es snapshot` and Kibana with `yarn
serverless-{mode}` where {mode} can be `es`, `security`, or `oblt`.
2. Verify that the Advanced settings app is not accessible and its
endpoint (`app/management/kibana/settings`) leads to the Stack
Management landing page.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR introduces two changes:
(1) Refactors the handler resolution logic to _not_ depend on the
`--serverless` cli arg by adding a new piece of config
`server.versioned.routeResolution` that accepts `newest | oldest`. This
piece of config is passed down instead of the `serverless` cli arg as
well as updating test cases
(2) Adds a new piece of config to turn off the client version checking.
This will be needed for rolling upgrades to allow old browser traffic to
reach new Kibana servers when there is stack version change.
Close https://github.com/elastic/kibana/issues/158723
## Open questions
* Do we want to make the version check still take _major_ version bumps
into account?
## Summary
Until [this issue](https://github.com/elastic/kibana/issues/153720) is
resolved, this config flag allows us to access the task manager
background worker utilization API in serverless to support autoscaling
of background task deployments
## To Verify
Run es: `yarn es snapshot`
Run serverless on this branch: `yarn serverless-es`
Verify you see the following warning in the logs:
```
[2023-06-12T12:47:19.641-04:00][WARN ][plugins.taskManager] Disabling authentication for background task utilization API
```
and you can access `/api/task_manager/_background_task_utilization`
without logging in
# Summary
This PR adapts the endpoint RBAC to the new Serverless PLI features
architecture.
The changes are the following:
## App Features
### New appFeatures keys for endpoint
The `endpointExceptions` PLI has been added to the _Endpoint Essentials_
product tier and `endpointResponseActions` to the _Endpoint Complete_
686bc2eeaa/x-pack/plugins/serverless_security/common/pli/pli_config.ts (L20-L23)
### Endpoint appFeatures capabilities config
The features configuration for each appFeature (PLI) has been added.
They will be configured within the Security Kibana features only when
the appFeature is enabled by the selected Security product type. (Note
that all of them will be always added in regular ESS deployments, only
in Serverless we'll have different product types)
4d9f0c3a6f/x-pack/plugins/security_solution/server/lib/app_features/security_kibana_features.ts (L170-L198)
These are the capabilities that seemed relevant to me for each PLI, but
I don't have enough expertise in Endpoint operations to know for sure
what Kibana sub-features and capabilities need to be included in each
appFeature. The PLIs are in a private spreadsheet with the following
descriptions.
- endpointExceptions:
- endpointResponseActions:
I'll need Endpoint team members to confirm there's no missing or wrong
capability in each appFeature config.
### Host isolation capabilities
It is important to mention that in the configuration above, to have some
capabilities available we are adding some sub-features directly using
the `subFeatureIds` entry, but for host_isolation capabilities, we are
doing it in a slightly different way, using the `subFeaturesPrivileges`,
this way the privileges are added to existing subFeatures.
The reason is we need to have the _write_ (isolate operation) only in
payment product types, but the _read_ and _delete_ (release operation)
capabilities should be always available, to allow releasing previously
isolated hosts after a product downgrade.
To do this we always include the `host_isolation_all` and
`host_isolation_exceptions_all` subFeatures in the base configuration,
but they only contain _read_ and _delete_ capabilities by default, only
when the product tier allows the proper appFeatures the _write_
capability is added to the same subFeatures privileges.
## Endpoint Authz module
### Remove "superuser" specific check
This specific check:
```
// user is superuser, always return true
if (isSuperuser) {
return true;
}
```
Has been removed, this has no behavioral impact, superuser has all
capabilities enabled anyway.
### Remove usage of `endpointRbacEnabled` and `endpointRbacV1Enabled`
experimental flags
They are already enabled by default. superuser will still have the
authorization to access all the features. The only change is the
endpoint sub-features will always be visible in the Kibana Privilege
section of the Role management page, they were hidden when these
experimental flags were disabled.
### Remove double _write_ check for _read_ authorizations:
We were doing unnecessary checks for the _write_ capabilities in the
_read_ authorizations, like: ```
const canReadEndpointList = canWriteEndpointList ||
hasKibanaPrivilege(fleetAuthz, 'readEndpointList');
```. Sub-features already add _read_ and _write_ capabilities on the
`all` privilege, so these double checks were unnecessary.
### Extract `hasHostIsolationExceptionsItems` flag
This flag was used to grant _read_ and _delete_ authorization for Host
Isolation Exceptions (HIE) when there is data, basically turning them
free features when there is data to perform the actions. This is needed
to allow users to remove HIE after a license downgrade scenario, which
is good.
However, we needed to do this API call from outside the auth module, in
every place we needed to call `calculateEndpointAuthz`, and we were also
adding the responsibility to do some auth-specific logic with licenses
outside the auth module, which is not good.
In addition, it is not very consistent to make authorization depend on
the existence of data to perform an action. Authorization should be
based only on the role capabilities and tiers/licenses, if some parts of
the application want to show/hide stuff depending on the data, that's
not the auth module's responsibility.
I checked all the places where we use the HIE _read_ and _delete_
authorizations, and the only place where we really need them to be
denied (when there is no data) is in the _links_, we need to remove the
HIE link from the app in this situation.
So, this PR moves the data check to the links.ts module, making the
_read_ and _delete_ permissions always granted without a license (they
will still be useless without data), the same way the `canUnIsolateHost`
authorization works. And then doing the async data check to remove the
HIE link in the _management/links.ts_ module itself, only in the last
case where we really need to know it:
4d9f0c3a6f/x-pack/plugins/security_solution/public/management/links.ts (L257-L262)
This flag extraction is unrelated to the integration of the new
architecture, I included it only to extract complexity from the _authz_
module and simplify its usage, but this change can be rolled back if we
consider it.
# Testing
- To start the application in ESS (non-serverless) mode, run it normally
with `yarn start`. Everything should keep working as usual with all
features available and capabilities should only be restricted by the
user role.
- To start the application in Serverless mode run with `yarn
serverless-security`. It sets a random root path, so access the main URL
at "http://localhost:5601/" to be redirected.
By default the "Endpoint Complete" product line is selected in the
_serverless.security.yml_ config, so everything should be available as
in ESS with the default config.
686bc2eeaa/config/serverless.security.yml (L11-L15)
Once in Serverless mode, in order to see the difference between product
types, we can change the _Endpoint_ `product_tier` to `essentials`, as
per the pli_config, this change should remove all the capabilities
included by the `endpointResponseActions` appFeatures config.
To check how the application behaves without the `endpointExceptions`
PLI, we can remove the _Endpoint_ `product_line` entirely from the
product array, leaving the _Security_ `product_line` alone.
# Next steps
## Upselling page
The product upselling page has not been registered for endpoint pages in
this PR, so when any of these pages are unauthorized because of the
serverless product tier, and they are accessed directly by URL they
still show the `Privileges required` screen.
This is arguably not entirely correct. However, an upselling page can be
registered to display a "Buy a higher tier" message when the privilege
is denied because of the product type, if it is unauthorized because of
the user role the "Privileges required" page will still show.
I did not include the endpoint upselling page in this PR to keep it
simple, but the registry is already implemented in the main proposal, we
can define and register them in a follow-up PR.
## Superuser role in authz module
Almost all "superuser" role conditionals have been removed from the
Endpoint authz module, but there is only one check left here:
24330f2356/x-pack/plugins/security_solution/common/endpoint/service/authz/authz.ts (L85)
This `canAccessEndpointManagement` flag looks deprecated, and it seems
to be used incorrectly in the few places where it is checked. If we
could fix the places that it is used, checking the proper authz flag, we
could definitively remove the `userRoles` parameter from the
`calculateEndpointAuthz` function, this will have an impact in the
different places where this function is called since they will no longer
need any async logic.
---------
Co-authored-by: Pablo Neves Machado <pablo.nevesmachado@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Closes https://github.com/elastic/kibana/issues/155371
## Summary
PR adds Serverless Onboarding flow using Custom Integration. This would
also lay the foundation for us to complete get rid of Home Tutorial App
and move the remaining `onPrem` and `cloud` tutorials which are
currently still loaded using Home Tutorial App.
1. Adds new Custom Integration for Serverless Onboarding (Toggling Home
AApp Tutorial Integration)
2. Since we are migrating away from the Home App Tutorials, lot of
existing code has been duplicated and refactored for the custom
implementation. Home App Tutorial would require the Server to register
all the steps and the client to only register a custom component which
then would be loaded by Home App Tutorial component. We don't need to
follow this approach any more. All the UX logic has now been moved to
the Public folder with only Custom Integration done on the
`server/plugin.ts`.
3. As we are not sure how the solutions will be informed about being
running on Serverless or not, I have introduced a new variable in
`serverless.oblt.yml` file called `xpack.apm.serverlessOnboarding:
true`. With this the development has been done. This can be changed to
actual logic once we know more.
4. A new configuration `xpack.apm.managedServiceUrl` for accessing
Managed Service URL is also being added by Control Plane team as part of
https://elasticco.atlassian.net/browse/CP-2403. Hence this PR expects
this property to be present for Serverless.
5. Unit tests to toggle between `secret_token` and `api_key` depending
on availability has been added. No API Tests were added as no new API
created. Cypress Tests cannot be added due to Serverless
## Need help reviewing the PR ?
1. `config/serverless.oblt.yml` - Adds the new flag which would enable
this flow
2. `x-pack/plugins/apm/common/tutorial/tutorials.ts` - Defines the
configuration required to register the APM's Tutorial Custom Integration
3. `x-pack/plugins/apm/public/components/app/tutorials/commands` - This
directory contains all the agent specific data required to load the
TABLE with settings required for configuring APM MIS.
4. `x-pack/plugins/apm/public/components/app/tutorials/instructions` -
This folder contains all the individual agent specific instructions in
the format used by
[EuiSteps](https://eui.elastic.co/#/navigation/steps#complex-steps)
5. `x-pack/plugins/apm/public/components/routing` - Here we register our
custom route
6. Changes on the server side a quite small and they only register the
custom integration.
7.
`x-pack/plugins/apm/public/components/app/tutorials/serverless_instructions.tsx`
- This file currently defines all the logic for registering Serverless
instructions. We will soon have similar files for `onPrem` and `cloud`
instructions
### Risk Matrix
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| The flow depends on presence of a flag in `kibana.yml` file. | Low |
High | By default this flow will be disabled and would fallback to
traditional onboarding in absence of the flag. |
### Demo
d60f0610-1fea-4540-86f5-2d72ab97f640
### Updated Demo with Create API Button inside the table
e84d8d6c-a048-4638-9b63-45080feca90b
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Since the only way to reach logs onboarding is manually entering the url
it has been decided to enabled this plugin by default.
In serverless it was already enabled by default through the config.
Resolves#159028Resolves#159047
## Summary
Hide create spaces button when limit is reached.
## Screenshot
## Testing
1. Set the maximum number of allowed spaces to 1
```yml
xpack.spaces.maxSpaces: 1
```
2. Verify that the create spaces button is hidden and that a callout is
displayed
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fixes: #155154 (introduced in #149878), builds on #155436 .
- Adds tests to ensure the configuration merging order, check those for
reference.
- Updates the README to explain the intention
For the tests, I needed to output something to the logs. I hope it's not
a big issue to log it. If needed, I might hide that behind a verbose- or
feature flag.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
issue: https://github.com/elastic/kibana/issues/158810
follow-up of: https://github.com/elastic/kibana/pull/158179
Improves the Security AppFeatures architecture:
- SubFeatures now preserve always the same order in the Security Kibana
config, despite the order of processing of enabled appFeatures.
- Change the `productTypes` config format
- Update `getProductAppFeatures` to:
- process the new `productTypes` format.
- include _essentials_ tiers PLIs inside _complete_ tiers automatically.
- AppFeatures module now receives an array of PLIs instead of an object
- AppFeatures config now uses only SubFeature IDS instead of
`subActions` config objects directly
- Upselling components updated and `useProductTypeByPLI` implemented to
display the Product Type required
[Documentation](https://docs.google.com/document/d/1Ms8d8d_fbTTRHlBroEAKGNMNk3jFFgOAkVDRhqLxAPQ/edit?pli=1#)
issue: https://github.com/elastic/kibana/issues/158810
## Summary
This PR is a cleanup to make [this
POC](https://github.com/elastic/kibana/pull/155420) production ready
- Serverless PLI features splitting in Security Solution, to allow/deny
access to configured functionalities, using the current Kibana RBAC
service.
- Create the Upselling service to display Serveless-specific prompts in
the application when features are not available
- Create a `SecurityRoutePageWrapper` component that wraps Pages and
displays the upsell when necessary.
- We will refactor the code base to use `SecurityRoutePageWrapper`
everywhere on another PR.
- Create an Upsell page and section for entity analytics
bd8db822-2f4b-4545-9da7-bedc07d93f90
### test:
Serverless: `yarn serverless-security`.
* To change the product line you have to update
`xpack.serverless.security.productLineIds` on
`config/serverless.security.yml`.
ESS: `yarn start`
### Glossary
* PLI - Product Line Item (`Alert Triage`, `Osquery`, `Cases` , ... )
* Product Line - The product that the user is subscribed to (Security
Essentials, Security Complete, ...)
* essSecurity - New plugin with code that only runs for ESS offer
(non-serverless).
* App Feature - A security solution feature or group of features that
can be disabled for a product line. It can be mapped to PLIs (`Alert
Triage`, `Osquery`, `Cases` , ... ).
* Capability - A string that when present represents that the user can
access a given feature. A capability could be of the type UI or API
(`read_cases`, `crud_cases`, ...).
### Current architecture
### New architecture
### How does it work?
Every serverless product line (endpointEssentials, cloud essentials) can
define which features are enabled:
69d0fc15f4/x-pack/plugins/serverless_security/common/pli/pli_config.ts (L12-L19)
For ESS (non-serverless) offer we enable all features by default.
69d0fc15f4/x-pack/plugins/ess_security/server/constants.ts (L10-L13)
A feature can define privileges:
69d0fc15f4/x-pack/plugins/security_solution/server/lib/app_features/security_kibana_features.ts (L177-L185)
When the feature is enabled the privileges get merged into the base
config and injected into kibana features.
69d0fc15f4/x-pack/plugins/security_solution/server/lib/app_features/app_features.ts (L61-L70)
### TODO
- [x] lazy load these components
- [x] Add unit test to:
- ~SecurityRoutePageWrapper
x-pack/plugins/security_solution/public/common/components/security_route_page_wrapper/index.tsx~
-
~x-pack/plugins/security_solution/public/common/hooks/use_upselling.ts~
-
~x-pack/plugins/security_solution/public/common/lib/capabilities/has_capabilities.ts~
-
~x-pack/plugins/security_solution/public/common/lib/upsellings/upselling_service.ts~
- ~x-pack/plugins/serverless_security/common/pli/pli_features.ts~
-
~x-pack/plugins/serverless_security/public/components/upselling/register_upsellings.tsx~
-
~x-pack/plugins/security_solution/server/lib/app_features/app_features.ts~
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Partially addresses https://github.com/elastic/kibana/issues/157756
## Summary
This PR disables the Search Sessions plugin for serverless.
**How to test:**
1. Start Elasticsearch with `yarn es snapshot` and Kibana with yarn
`serverless-{mode}` where `{mode}` can be `es`, `security`, or `oblt`.
2. Verify that the Search Sessions app is not accessible and its path
(`app/management/kibana/search_sessions`) leads to the Stack Management
landing page.
## Summary
This PR makes the Migrate plugin disable-able for serverless.
Partially addresses https://github.com/elastic/kibana/issues/157756
**How to test:**
1. Start Elasticsearch with `yarn es snapshot` and Kibana with yarn
`serverless-{mode}` where `{mode}` can be `es`, `security`, or `oblt`.
2. Verify that the Migrate plugin doesn't show up in the nav bar and its
path (`management/data/migrate_data`) leads to the Stack Management
landing page.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Closes#150884
## Summary
Adds both strictTransportSecurity and disableEmbedding to the
serverless.yml config file, which sets the defaults for Kibana running
in a serverless configuration. Note that the disabling of embedding is
for the serverless MVP, with a plan to more strategically and explicitly
allow embedding of specific resources in future releases.
- `server.securityResponseHeaders.strictTransportSecurity:
max-age=31536000; includeSubDomains`: Adds the
`strict-transport-security` response header. The browser should remember
that a site, including subdomains, is only to be accessed using HTTPS
for 1 year.
- `server.securityResponseHeaders.disableEmbedding`: true adds the
`X-Frame-Options` response header with a avalue of `SAMEORIGIN` and adds
`frame-ancestors 'self'` to the `content-security-policy` response
header.
Note: if you are running without TLS enabled locally, you can disable
the `strict-transport-security` response header by overriding the
setting in kibana.dev.yml (see Testing step 8 below).
### Testing (locally)
1. Start Elasticearch
3. Start Kibana with `yarn start --serverless`
4. Sign into Kibana and open your browser's dev tools
5. In the network tab, inspect one of the requests for localhost
6. In the Headers tab, verify the above defined headers and values are
present
7. Stop Kibana, and restart normally with `yarn start`
8. Repeat the process in steps 4-6 and verify that the above defined
headers and values are not present
9. Stop Kibana, edit the Kibana.dev.yml file by adding
`server.securityResponseHeaders.strictTransportSecurity: null`
10. Start Kibana with `yarn start --serverless`
11. Repeat the process in steps 4-6 and verify that the headers and
values associated with disableEmbedding are present while the
`strict-transport-security` response header is not present.
## Summary
- Updates the project navigation to match latest design with a single
combined left nav
- Disables & removes `enterprise_search` plugin usage as we are not
linking to any ent-search pages for this milestone
*Note: the "Getting started" nav item will always show as active until
the Navigation component is updated to determine the selected nav item,
which will be done with other upcoming work.
## Screenshots
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
The PR introduces the first iteration of the side navigation bar for
serverless observability. Part of
https://github.com/elastic/kibana/issues/153777
### Run locally
```
yarn serverless-oblt
```
#### Screenshots
39ded143-0d4b-4ea6-9534-0ca87b80662d
### Fixes
- Fix rendering the icon for the nested navigation items
-
24ee4dc616
### Notes
- There is an issue where the selected navigation item is not properly
highlighted and loses focus when the user clicks anywhere else on the
page. (atm out of the scope of the PR)
- The navigation tree is subject to change as there is an ongoing
discussion about the naming and order
---------
Co-authored-by: Søren Louv-Jansen <soren.louv@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
> Derived from https://github.com/elastic/kibana/pull/153274
## Summary
This PR extracts configuration settings for enabling/disabling plugins
in Serverless projects based on current requirements.
It seemed prudent to create an independent PR to K.I.S.S, rather than
include in PRs with more ornate changes, (e.g.
https://github.com/elastic/kibana/pull/155582)
## 📓 Summary
Closes#153890
The implementation creates a new LogsApp service where we should keep
any logic concerned with what `target_app` parameter is configured and
the actions related to a specific configuration. I thought it could be a
good approach to avoid drilling down the global config till we need it
and keep it cleaner by injecting only the service with predefined
actions.
In this first case, we create a redirect to discover using its locator,
and the exposed method can be used anywhere across the app for
triggering the redirect.
## 🧪 Testing
### Normal behaviour
When Kibana is used as always, we want to keep the current behaviour and
the user will stay on the Logs UI pages.
- Launch the Kibana dev environment with `yarn start`
- Navigate to Logs UI
- Verify the navigation works normally and that no redirect to Discover
occurs
### Serverless behaviour
When Kibana is used in serverless mode, we want to redirect any user
landing to Logs UI to the Discover page, configuring the same data view
or creating an ad-hoc one starting from the index pattern
- Launch the Kibana dev environment with `yarn serverless-oblt`
- Navigate to Logs UI
- Verify to be redirected to Discover and a temporary data view is
created from the current index pattern
---------
Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
We are currently expecting serverless environments to be broken up into
several different envs, this config structure allows us to customize the
config based on that environment without major modifications to the
config loading system.
This adds the `--serverless` CLI arg (only available in dev mode, you
can also use `yarn start-serverless`), which will load the new
`config/kibana.serverless.yml` file. For now, this file is not included
in the build artifact, though we might include a `--serverless` flag
which replaces the `kibana.yml` file with `kibana.serverless.yml`.
@jbudz will follow up after this PR with the build related changes to
get this working with PR cloud deploys, which will be enough changes for
us to start iterating on UI specific changes based on running in a
serverless environment.
Additionally, support for the undocumented `KBN_CONFIG_PATHS` env var is
added, which should contain a comma-separated list of paths to kibana
config files. These files are loaded, in the specified order, before any
of the config files listed in the CLI but after the kibana.yml config
file.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* [config] Remove leading spaces from logging configuration
This is a consistency check with other areas of the stack and other
configurations in kibana.yml.
Part of https://github.com/elastic/kibana/issues/8268
* Update kibana.yml
* remove kbn-legacy-logging package
* remove legacy service
* remove legacy appender
* remove LegacyObjectToConfigAdapter
* gix types
* remove @hapi/good / @hapi/good-squeeze / @hapi/podium
* remove `default` appender validation for `root` logger
* remove old config key from kibana-docker
* fix FTR config
* fix dev server
* remove reference from readme
* fix unit test
* clean CLI args and remove quiet option
* fix type
* fix status test config
* remove from test config
* fix snapshot
* use another regexp
* update generated doc
* fix createRootWithSettings
* fix some integration tests
* another IT fix
* yet another IT fix
* (will be reverted) add assertion for CI failure
* Revert "(will be reverted) add assertion for CI failure"
This reverts commit 78d5560f9e.
* switch back to json layout for test
* remove legacy logging config deprecations
* address some review comments
* update documentation
* update kibana.yml config examples
* add config example for `metrics.ops`
Co-authored-by: Tyler Smalley <tyler.smalley@elastic.co>
