Updates the text on the landing page
**Before**
**After**
Adds a new feature flag to allow the `Create filebeat configuration`
card to be hidden. This has been
[requested](https://github.com/elastic/enterprise-search-team/issues/7075)
for the serverless search project and so the `serverless.es.yml` has
been updated to include this flag.
**Before**
**After**
## Summary
This PR enables the `xpack.ml.nlp.enabled` flag for the security project
in support of enabling ELSER for use with the Elastic Assistant.
Reference issue: https://github.com/elastic/security-team/issues/7995
Corresponding `project-controller` PR's here:
https://github.com/elastic/project-controller/pull/674https://github.com/elastic/project-controller/pull/751
---
As discussed in the above reference issue, this PR adds a task for
cleaning up NLP models at a regular interval if the `productTier` is not
`complete`. This `serverless-security:nlp-cleanup-task` will run every
`6hr`, attempting to cleanup any `pytorch` NLP models/aliases that were
either leftover from a `productTier` downgrade, or were manually created
using the ES trained model API's.
Note I: As recommended by the ResponseOps folks, the task is _always
registered_ regardless of the current `productTier`. When executed on
the `complete` `productTier`, an `UnrecoverableError` is thrown, which
will remove the task and prevent it from running again until Kibana is
restarted.
Note II: While the task interval is every `6hr`, when initially
registered we manually kick off an execution to ensure that any
resources are immediately cleaned up when a `productTier` change is
performed.
---
Both `complete` and `essentials` API integration tests have been added
to cover the above behavior. Please see:
*
[.../test_suites/genai/nlp_cleanup_task/basic_license_essentials_tier/task_execution.ts
](https://github.com/elastic/kibana/pull/175358/files#diff-b0cae16c5c6f77a90d53fcfab3bda6400ec13fe7f200d8c1f0dd3f1a0109af9f)
*
[.../test_suites/genai/nlp_cleanup_task/trial_license_complete_tier/task_execution.ts](https://github.com/elastic/kibana/pull/175358/files#diff-c69ec08309855f236ab90d4ba73f40636e52d706679bbab530956ff8d5560097)
---
Additionally, Event Based Telemetry (EBT) was added so cleanup, and any
associated failures can be tracked. Please see the
`NLP_CLEANUP_TASK_EVENT`, which has the below schema:
```
failedToDeleteCount: number; // Total number of NLP models failed to be cleaned up
message: string; // General message from task completion, either summary of task or error message
productTier: string; // Current productTier when task was run
totalInstalledCount: number; // Total number of NLP models deployed when task started
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
### Summary
This PR enables the NLP models management page in the ML plugin for
Observability serverless projects and updates how the Obs AI Assistant
checks if ELSER is ready to be used.
## Summary
issue: https://github.com/elastic/kibana/issues/179162
This PR brings the new navigation, with the solution-centric IA
(Information Architecture), to the ESS (stateful) Security Solution.
To do so, the implementation of the `navigationTree$`, which was
previously only implemented in serverless, has been integrated inside
the generic "security_solution" plugin, so now it is available for ESS
and serverless offerings.
In ESS users can still choose the navigation version, so we have to
temporarily keep supporting both, the classic and new navigation
implementations. After the rollout, the classic navigation components
will be removed and the unified links architecture should be reassessed.
The issue for the cleaning:
https://github.com/elastic/kibana/issues/179572
### Rollout
The new solutions navigation will not be available for customers on
8.14, it will only be enabled for internal Elastic users (via
Lauchdarkly), who will have the ability to opt out of it from their
profile menu. We'll collect feedback and telemetry and address any bugs
or improvements (together with the Kibana platform team). The plan is to
start making it available to customers in 8.15.
### Testing
Unless we add the _kibana.yml_ configurations to enable the new
navigation, the regular classic navigation will be displayed, it should
keep working the same way without any change. The new landing pages
(`Assets`, `Investigations`, `Machine Learning`,...), that exist only
when using the new navigation, should not be accessible using the
classic version.
To enable the new navigation add the following _kibana.yml_ configs:
```
xpack.cloud_integrations.experiments.enabled: true
xpack.cloud_integrations.experiments.flag_overrides:
"navigation.solutionNavEnabled": true
xpack.cloud.id: "ftr_fake_cloud_id:aGVsbG8uY29tOjQ0MyRFUzEyM2FiYyRrYm4xMjNhYmM="
xpack.cloud.base_url: "https://cloud.elastic.co"
xpack.cloud.deployment_url: "/deployments/deploymentId"
```
And enable the advanced setting
### Screenshots
The app switcher:
<img width="293" alt="app switcher"
src="0a638b8f-fdc0-4d1a-b8d3-607e487215f4">
---
New Assets landing page:
<img width="549" alt="assets landing"
src="17bc8a94-02b4-4996-b9f5-8731ba81ac43">
---
For `Stack Management` we set the nav panel flyout and the cards landing
page, this is temporary until a decision on how to show Stack Management
links is made:
<img width="954" alt="stack management"
src="27ce6534-0508-4804-b224-8dc409042825">
---
The switch to go back to the classic nav is in the profile menu at the
top-right corner:
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Søren Louv-Jansen <soren.louv@elastic.co>
Co-authored-by: Vitalii Dmyterko <92328789+vitaliidm@users.noreply.github.com>
Co-authored-by: jennypavlova <dzheni.pavlova@elastic.co>
Co-authored-by: Katerina <aikaterini.patticha@elastic.co>
Co-authored-by: Sébastien Loix <sebastien.loix@elastic.co>
Co-authored-by: Kurt <kc13greiner@users.noreply.github.com>
Co-authored-by: Justin Kambic <jk@elastic.co>
Co-authored-by: Julia Bardi <90178898+juliaElastic@users.noreply.github.com>
Co-authored-by: Paul Tavares <56442535+paul-tavares@users.noreply.github.com>
Co-authored-by: Nathan Reese <reese.nathan@elastic.co>
Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com>
Co-authored-by: Milton Hultgren <milton.hultgren@elastic.co>
Adds a setting in the `aiAssistantManagementSelection` plugin that
allows users to set the preferred Assistant type. The allowed values
are:
- `default`: the status quo, which is: show the Observability AI
Assistant in Observability apps, the Security AI Assistant in Security
solution apps, and none in the other apps.
- `never`: Never show any AI Assistant.
- `observability`: Show the Observability AI Assistant everywhere,
except for Security solution apps.
- `security`: ~Show the Security AI Assistant everywhere, except for
Observability apps.~
it is up to the solutions to respect the setting. See
[x-pack/plugins/observability_solution/observability_ai_assistant_app/public/hooks/is_nav_control_visible.tsx](https://github.com/elastic/kibana/pull/179233/files#diff-753a9aae298da45c5bd96bc2cacaa437ac53e165069ad6323973dd3c2879d4ae)
for an example. It should also be set in the specific Serverless config:
[config/serverless.oblt.yml](https://github.com/elastic/kibana/pull/179233/files#diff-358680d2b72268283dca4f3efb976cd924b7b5bce822e07c8871d061b4bb339e).
Update: this is now labeled as an Observability-only setting.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Disable the Obs AI Assistant plugin in Serverless Search & Serverless
Security. Some work was done to move `observabilityAIAssistant` from
`requiredPlugins` to `optionalPlugins` in upstream dependencies.
Closes https://github.com/elastic/kibana/issues/166999
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fix https://github.com/elastic/kibana/issues/178180
- Add a new `core.lifecycle.disablePreboot` (internal) config option to
forcefully disable Core's `preboot` phase.
- Enable the option in the serverless configuration file
Gain is around 150/200ms on local developer machine, which translates to
~300/500ms on serverless environment
Fixes https://github.com/elastic/kibana/issues/176420
## 🍒 Summary
This PR copies the SLO code that was inside the Observability app into
its own app under `observability-solution/slo` folder.
4f6b8dfb-9612-4d30-ad50-4ee5c55a9c32
## ✔️ Acceptance criteria
- URL of new app: `app/slos`
- Design and functionality are not changed.
- Git history has been retained for all files in
`x-pack/plugins/observability_solution/slo`.
- SLO should appear on server less
- SLO code inside `observability_solution/observability` code has been
removed. A new clean up round might be needed though for possible
leftovers.
- Burn rate rule is registered within the new slo app
- SLO embeddables are moved inside the new slo app
- overview
- alerts embeddable
- error budget burn down
- Alerts table configuration registration for slo details page and
alerts table embeddable is still done in the observability app. Response
Ops team is working on removing the need to register the alert table
anyway
- Slo app is wrapped into `ApplicationUsageTrackingProvider` which will
send slo `Application usage` information tracked by the `slo` appId
- Redirect old `app/observability/slos` route to `app/slos`
- Rename old `xpack.observability.slo` keys to `xpack.slo` in the
translation files
## 🌮 How to test
Design and functionality didn't change, so simply navigate to existing
slo pages and try to break it
- Slo list page
- group by
- unified search
- toggle buttons
- actions
- Slo creation
- try group by as well
- Slo detail page
- Actions on top
- navigate to overview and alerts tabs
- Create SLO flyout in Logs Explorer
- Create burn rate rules and verify they appear on rules page
- Verify SLO alerts appear on Alerts page and slo details page
- Embeddables
- Through the dashboard app
- Using the attach to dashboard action on the slo card item on slo list
page and the error budget burn down chart on the slo detail page
- SLOs only for platinum users
- Permissions
- Spaces
## TODO
- [x] Move slo stuff from observability folder to new slo plugin
- [x] Remove old slo stuff from observability folder
- [x] Update references
- [x] Fix typescript and eslint errors
- [x] Paths
- [x] Locators
- [x] Burn rate rule registration
- [x] Embeddable Alerts table configuration registration
- [x] Embeddables
- [x] Translations
- [x] Verify plugin.ts files contain all registration logic
- [x] public
- [x] server
- [x] Final cleanup for observability folder
- [x] Run tests
- [x] Application Usage (Telemetry)
- [x] Permissions
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: shahzad31 <shahzad31comp@gmail.com>
Co-authored-by: Coen Warmer <coen.warmer@gmail.com>
Towards: #176585
This PR removes the task skipping logic from TaskManager, PRs for
Alerting and Actions will follow.
## To verify
Rules and actions should be still working as expected.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
At present, we have the MKI Kibana Controller disable Synthetics in
production Serverless by default.
This patch will make Synthetics enabled by default, so at the point when
we want to remove the injected disable config on the controller side,
Synthetics will be available to all production Observability projects
immediately.
## Summary
Closes https://github.com/elastic/kibana/issues/174771
While the security team works on Custom Roles for serverless, we want to
hide the content behind a feature flag.
An existing config option that was used to hide the Roles UI during the
initial phases of serverless has been repurposed, and will now toggle
both the Roles UI and the Roles Routes
`xpack.security.confg.ui.roleManagementEnabled` has been changed to
`xpack.security.confg.roleManagementEnabled` and will have to be set to
`true` in a config file while in serverless mode to show the Roles card
on the management screen and enable the UI/routes.
## Reviewers
Ive included a `viewer`:`changeme` user for testing (It will be removed
after approval).
## Testing
### xpack.security.config.roleManagementEnabled
1. In your `kibana.yml`, add
`xpack.security.confg.roleManagementEnabled: true`
2. Start up in serverless mode locally, login in with
`elastic_serverless`:`changeme`
3. Click `Project Settings` > `Management`
4. `Roles` card should display under `Other`
5. Navigate to `Roles`, it displays, but the `Edit Roles` page does not
work yet.
### Test as Viewer
1. In your `kibana.yml`, add either above option as you prefer
2. Start up in serverless mode locally, login in with
`viewer`:`changeme`
3. Click `Project Settings` > `Management`
4. `Roles` card should NOT display under `Other` and the `roles` URL
should not work.
## Screenshots
Roles card
<img width="1281" alt="Screenshot 2024-02-05 at 3 22 12 PM"
src="a1285ada-7ff7-495f-88a6-9847b3245518">
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>
## Summary
Part of https://github.com/elastic/kibana/issues/144276
- Introduce the concept of browser-side logging configuration, via a
`logging.browser` config prefix
- Allow to configure the log level for the root browser logger via
`logging.browser.root.level`
- Set the default level to `info` for both dev and production mode
(consistent with server-side logging)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Resolves: https://github.com/elastic/kibana-team/issues/721
As we discovered by an issue, it's an anti pattern to timeout-and-retry
requests part of the polling cycle because it's basically putting the
requests back in the same queue that it once was ahead on.
Therefore we would like to make TaskManager's update-by-query request
timeout configurable.
Closes https://github.com/elastic/kibana/issues/165672
### Summary
This PR adds support for custom output and Fleet server hosts with
proxies in serverless mode:
- Proxies are re-enabled in serverless.
- It is possible to add custom Fleet server hosts in serverless, with
the constraint that the host URL must match the Elasticsearch URL of the
default host.
- New Elasticsearch outputs must also have the default host URL.
### Testing
The below requirements should be tested in serverless mode for
observability and security project types:
```sh
# elasticsearch
yarn es serverless --kill
# kibana: one of
yarn serverless-oblt
yarn serverless-security
```
⚠️ In addition, stateful mode should **not** be affected by any of these
changes.
#### Config
In order to test this change, you will need the following configuration
to mirror a serverless setup.
Create a `serverless.dev.yml` if you don't have one already and set a
project id (this is required for Fleet's `cloud.isServerlessEnabled` to
correctly be `true`):
```yaml
xpack.cloud.serverless.project_id: test-123
```
In `kibana.devl.yml`, make sure the default Fleet server hosts and
default output have the expected ids:
```yaml
xpack.fleet.fleetServerHosts:
- id: default-fleet-server
name: Default Fleet server
is_default: true
host_urls: ['https://host.docker.internal:8220']
xpack.fleet.outputs:
- id: es-default-output
name: Default output
type: elasticsearch
is_default: true
is_default_monitoring: true
hosts: ['https://host.docker.internal:9200']
```
#### Requirements
- [ ] User can create proxy configurations in the Fleet UI and API.
- [ ] User can create new Fleet server host via the UI
- The Fleet Server Hosts dropdown is disabled and set to the default
host URL
- [ ] User can create new Fleet server host via the API
- The request should succeed if the host URL is set the the default one
- Otherwise the request should fail
- [ ] User can add a proxy config to the Fleet server host config
- [ ] User can select a custom Fleet Server host configuration from the
Agent policy settings page
- [ ] User can create new Fleet Elasticsearch output via the UI
- The Hosts dropdown is disabled and set to the default Elasticsearch
URL
- [ ] User can create new Fleet Elasticsearch output via the API
- The request should succeed if the host URL is set the the default one
- Otherwise the request should fail
- [ ] User can add a proxy config to the Elasticsearch output
- [ ] User can select a custom Elasticsearch output configuration from
the Agent policy settings page
- [ ] User can create a custom Logstash output with proxy
- [ ] User can create a custom Kafka output with proxy
### Checklist
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### Screenshots
Fleet settings (proxies available):
Adding and editing a Fleet server host:
Adding and editing an Elasticsearch output:
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Introduced the console.ui.embeddedEnabled (boolean) value to allow
disabling the embedded console in serverless security and observability
projects.
This also replaces an index management specific config value to do that
same thing which is removed here as well. Adding this console
configuration will allow us to have one value to set instead of N which
was the pattern I had started to use, but then decided against.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
Closes https://github.com/elastic/kibana/issues/172922
## Summary
This PR:
- Integrates the new Settings application
(`packages/kbn-management/settings/application`) into stateful Kibana
and removes the old `management_app` from the
`src/plugins/advanced_settings` plugin.
- Adds support for section registry in the new Settings application, so
that other plugins can add their own sections to the Advanced settings
app.
- Adds functionality for disabling saving of settings based on the
provided capabilities of the current user.
<img width="1352" alt="Screenshot 2024-01-23 at 16 46 03"
src="1f3b7088-58e2-46e8-a7dd-ae0fc346b4ba">
<br><br>
"Usage collection" section in Global settings:
<img width="1099" alt="Screenshot 2024-01-23 at 16 48 24"
src="ebc54ad5-348b-46dd-a047-b418ddc7ba4f">
### How to test
**Testing Advanced settings in stateful Kibana:**
1. Start Es with `yarn es snapshot` and Kibana with `yarn start`
2. Go to Stack Management -> Advanced Settings
3. Verify that the app functions correctly. Both tabs (for space and
global settings) should be displayed, setting fields should be editable
and saveable, etc.
**Testing the section registry**
Currently, `telemetry_management_section` is the only plugin that
registers a section - the "Usage collection" section under the "Global
settings" tab. This should work correctly in stateful Kibana.
1. Start Es with `yarn es snapshot --license=trial` and Kibana with
`yarn start`
2. Go to Stack Management -> Advanced Settings and select the "Global
settings" tab
3. Scroll down and verify that the "Usage collection" section is
displayed and works as expected.
**Testing with different capabilities:**
1. Start Es with `yarn es snapshot` and Kibana with `yarn start`
2. Go to Stack Management -> Roles
3. Create a role that has "Read" access to Advanced settings and one
that doesn't have any access.
4. Create users with each of these two roles.
5. Log in with these users and verify that the user with "Read" access
can see the app but cannot edit it, and the user with no privileges
cannot access the app.
**Testing Advanced settings in serverless Kibana:**
The Advanced settings app in serverless shouldn't be affected by these
changes.
1. Start Es with `yarn es serverless` and Kibana with `yarn
serverless-{es/oblt/security}`
2. Go to Management -> Advanced Settings
3. Verify that the app functions correctly. There shouldn't be any tabs
as there are no spaces.
<!--
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
-->
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
We will depend on the apm-data Elasticsearch plugin for setting up index
templates and ingest pipelines. We have been testing this in serverless
dev and QA with config overrides -- this is just final step to roll it
out to all environments.
See https://github.com/elastic/apm-server/issues/11529
- Rename `log_explorer` to `logs_explorer` and move it to an
`observability_solution` subfolder
- Rename `observability_log_explorer` to `observability_logs_explorer`
and move it to an `observability_solution` subfolder
- Use `renameFromRoot` to rename old configs
Related to https://github.com/elastic/kibana/issues/171991 and
https://github.com/elastic/kibana/pull/170759
This PR is mostly focused on renaming the plugins, follow ups will be
created to change mentions of `log explorer`, `logExplorer` etc.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani01@gmail.com>
## Summary
Add an example logging config snippet that includes sensible defaults
(in the example) for log rotation, for when users uncomment and want to
point logging to a different path or adjust log to have file rotation
and start with an example.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Resolves https://github.com/elastic/synthetics-dev/issues/290.
~Creating this PR as a place to experiment with Synthetics running
against the Serverless platform. Main goals are to find areas of the
codebase that require revision and get as much of Synthetics functioning
as possible without access to public locations.~
This adds the necessary config and other features to make Synthetics
workable as part of the oblt serverless project. Notably, we aren't
including the `xpack.uptime.enabled` flag here, because we are not ready
to expose the plugin to production users yet. We're going to enable the
plugin on a per-env basis using other means to start, and when we are
ready to expose the plugin generally in prod, we will add that flag.
Also adds nav and fixes a few other things that were broken for
serverless specifically.
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Summary
In this PR we:
* Allow using JWT credentials to grant API keys
* Extend default value of `elasticsearch.requestHeadersWhitelist` to
include both `authorization` and `es-client-authentication` to support
JWT with required client authentication _by default_. See
https://www.elastic.co/guide/en/elasticsearch/reference/8.11/jwt-auth-realm.html#jwt-realm-configuration
* Add API integration tests for both JWTs with client authentication and
without it
__NOTE:__ We're not gating this functionality with the config flag
(`xpack.security.authc.http.jwt.taggedRoutesOnly`) as we did for the
Serverless offering. It'd be a breaking change as we already implicitly
support JWT authentication without client authentication, and to be
honest, it's not really necessary anyway.
## Testing
Refer to the `Testing` section in this PR description:
https://github.com/elastic/kibana/pull/159117.
Or run already pre-configured Kibana functional test server:
1. `node scripts/functional_tests_server.js --config
x-pack/test/security_api_integration/api_keys.config.ts`
2. Create a role mapping for JWT user:
```bash
curl -X POST --location "http://localhost:9220/_security/role_mapping/jwt" \
-H "Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-d "{
\"roles\": [ \"superuser\" ],
\"enabled\": true,
\"rules\": { \"all\": [{\"field\" : { \"realm.name\" : \"jwt_with_secret\" }}] }
}"
```
3. Send any Kibana API request with the following credentials:
```bash
curl -X POST --location "xxxx"
-H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJodHRwczovL2tpYmFuYS5lbGFzdGljLmNvL2p3dC8iLCJzdWIiOiJlbGFzdGljLWFnZW50IiwiYXVkIjoiZWxhc3RpY3NlYXJjaCIsIm5hbWUiOiJFbGFzdGljIEFnZW50IiwiaWF0Ijo5NDY2ODQ4MDAsImV4cCI6NDA3MDkwODgwMH0.P7RHKZlLskS5DfVRqoVO4ivoIq9rXl2-GW6hhC9NvTSkwphYivcjpTVcyENZvxTTvJJNqcyx6rF3T-7otTTIHBOZIMhZauc5dob-sqcN_mT2htqm3BpSdlJlz60TBq6diOtlNhV212gQCEJMPZj0MNj7kZRj_GsECrTaU7FU0A3HAzkbdx15vQJMKZiFbbQCVI7-X2J0bZzQKIWfMHD-VgHFwOe6nomT-jbYIXtCBDd6fNj1zTKRl-_uzjVqNK-h8YW1h6tE4xvZmXyHQ1-9yNKZIWC7iEaPkBLaBKQulLU5MvW3AtVDUhzm6--5H1J85JH5QhRrnKYRon7ZW5q1AQ'
-H 'ES-Client-Authentication: SharedSecret my_super_secret'
....for example....
curl -X GET --location "http://localhost:5620/internal/security/me" \
-H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJodHRwczovL2tpYmFuYS5lbGFzdGljLmNvL2p3dC8iLCJzdWIiOiJlbGFzdGljLWFnZW50IiwiYXVkIjoiZWxhc3RpY3NlYXJjaCIsIm5hbWUiOiJFbGFzdGljIEFnZW50IiwiaWF0Ijo5NDY2ODQ4MDAsImV4cCI6NDA3MDkwODgwMH0.P7RHKZlLskS5DfVRqoVO4ivoIq9rXl2-GW6hhC9NvTSkwphYivcjpTVcyENZvxTTvJJNqcyx6rF3T-7otTTIHBOZIMhZauc5dob-sqcN_mT2htqm3BpSdlJlz60TBq6diOtlNhV212gQCEJMPZj0MNj7kZRj_GsECrTaU7FU0A3HAzkbdx15vQJMKZiFbbQCVI7-X2J0bZzQKIWfMHD-VgHFwOe6nomT-jbYIXtCBDd6fNj1zTKRl-_uzjVqNK-h8YW1h6tE4xvZmXyHQ1-9yNKZIWC7iEaPkBLaBKQulLU5MvW3AtVDUhzm6--5H1J85JH5QhRrnKYRon7ZW5q1AQ' \
-H 'ES-Client-Authentication: SharedSecret my_super_secret' \
-H "Accept: application/json"
----
{
"username": "elastic-agent",
"roles": [
"superuser"
],
"full_name": null,
"email": null,
"metadata": {
"jwt_claim_sub": "elastic-agent",
"jwt_token_type": "access_token",
"jwt_claim_iss": "https://kibana.elastic.co/jwt/",
"jwt_claim_name": "Elastic Agent",
"jwt_claim_aud": [
"elasticsearch"
]
},
"enabled": true,
"authentication_realm": {
"name": "jwt_with_secret",
"type": "jwt"
},
"lookup_realm": {
"name": "jwt_with_secret",
"type": "jwt"
},
"authentication_type": "realm",
"authentication_provider": {
"type": "http",
"name": "__http__"
},
"elastic_cloud_user": false
}
```
__Fixes:__ https://github.com/elastic/kibana/issues/171522
----
Release note: The default value of the
`elasticsearch.requestHeadersWhitelist` configuration option has been
expanded to include the `es-client-authentication` HTTP header, in
addition to `authorization`.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Addresses https://github.com/elastic/kibana/issues/168684
## Summary
This PR adds validations of the user input for a some of the uiSettings
(see https://github.com/elastic/kibana/issues/168684) based on the
`schema` that is provided in the setting definitions. We do this by
exposing a `validateValue` API from the uiSettings browser-side client,
which calls an internal validate route to the server. Then
`validateValue` is called in the `onChange` handlers of the setting
input fields, utilising `debounce` to reduce the number of requests to
the server.
Note: One of the validation requirements is for validating that the
`defaultIndex` setting value is an existing data view. This is a more
complicated validation and will be addressed in a separate PR.
### How to test
**Testing the validations in serverless:**
1. Start Es with `yarn es serverless` and Kibana with `yarn
serverless-{es/oblt/security}`
2. Go to Management -> Advanced settings
3. Verify that the UI provides correct validation error messages for
[the settings that require
validation](https://github.com/elastic/kibana/issues/168684).
**Verify that self-managed Kibana is not affected:**
There should be no validations in self-managed Kibana and setting an
invalid value for an advanced setting should be allowed and shouldn't
cause any errors.
@elastic/kibana-core team, I know you're no longer code owners of the
uiSettings service, but any feedback on those changes will be highly
appreciated. 🙏
<!---
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
-->
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Per https://github.com/elastic/security-team/issues/7803, the security
solution team would like to increase the timeout for the threat
indicator match rule to 10 minutes. They're experiencing timeouts with
the 1 minute circuit breaker and are comfortable with the trade-offs
this will have.
Monitoring of task consumption has shown the `reports:monitor` task
takes an aggressively high amount of cycles from Kibana Task Manager.
This impacts the throughput of alerts.
This first step is a serverless-only config change to give an immediate
increase in overall task throughput.
It's safe to lower this polling frequency: **the impact is limited to
the responsiveness of retries** when a report job is found to be timed
out. Long-term, the plan will be to tune other parts of the code:
https://github.com/elastic/kibana/issues/170462
## Testing
1. Adjust the Dev mode settings to match a value set in production. Add
this to `config/kibana.dev.yml`:
```
xpack.reporting.capture.maxAttempts: 3 # usually in Dev mode, this is
set to 1
```
2. Start the scripts in different terminal windows to run Elasticsearch
and Kibana dev servers
```
yarn es serverless
---------------
yarn serverless
```
3. Open a search in Discover that covers about 4,000 hits and request a
CSV export using the Share menu.
4. Monitor the Kibana server logs and wait until the background job
begins. Restart the server while the job is executing (saving a file in
code under `packages/` or `server/` will trigger a restart).
5. Around 3 minutes after restart, the report job will get a re-attempt
Compare this behavior vs in non-serverless, where a report job will get
a re-attempt around 3 seconds after restart.
## Summary
This PR removes reporting deprecation log messages in kibana server logs
on serverless environments.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Tim Sullivan <tsullivan@users.noreply.github.com>
Adds a new `tags` property to our ML Modules which contains an array of
strings to used to filter the modules returned when calling
`/internal/ml/modules/get_module` or `/internal/ml/modules/recognize`
Adds a new kibana config setting `xpack.ml.compatibleModuleType` which
will enforce a module filter.
This setting supports the values `security`, `observability` or `search`
and will be used by the serverless projects to ensure only modules
relevant to the current project are discoverable.
When matching against the `xpack.ml.compatibleModuleType` setting,
modules with no tags will be returned along with modules which match the
`compatibleModuleType` value.
The endpoints `/internal/ml/modules/get_module` and
`/internal/ml/modules/recognize` can also take a `filter` query
parameter in the form of a comma separated list of values. If any of
these filter values match a module's tags, the module will be returned.
Modules with no tags will not be matched when using the `filter`
parameter.
This PR also updates the security plugin to apply a `security` filter
when retrieving modules and running recognize. This will improve
performance, especially for recognize which runs the queries from every
module on the supplied index pattern.
Examples
**Running the Security serverless project.**
Request:
`/internal/ml/modules/get_module`
Response:
All modules containing `tags: ["security"]`
Plus all modules with no `tags` or `tags: []`
Request:
`/internal/ml/modules/get_module?filter=observability`
Response:
An empty list
**Running stateful es**
Request:
`/internal/ml/modules/get_module`
Response:
All modules
Request:
`/internal/ml/modules/get_module?filter=security`
Response:
Only modules containing `tags: ["security"]`
Request:
`/internal/ml/modules/get_module?filter=security,observability`
Response:
Modules containing `tags: ["security"]` and `tags: ["observability"]`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Branch with Infra UI enabled in Serverless.
This exists for us to have a working environment based on main, but with
infra turned on. We could merge this at a later date when we plan to
enable Infra on serverless.
Fixes https://github.com/elastic/kibana/issues/167850Fixes#168065Fixes#168853
Fixes https://github.com/elastic/kibana/issues/167654
## Summary
This PR removes the Storage size column in Data stream in serverless as
the data stream stats API on serverless doesn't currently return the
storage size.
### How to test:
Verify that the column is not displayed in serverless:
1. Start Es with `yarn es serverless` and Kibana with `yarn
serverless-{es/oblt/security}`
2. Go to Stack Management -> Index Management -> Data Streams
3. Switch on the "Include stats" toggle
4. Verify that the "Storage size" column is not shown in the table.
Verify that the column is displayed in stateful:
1. Start Es with `yarn es snapshot` and Kibana with `yarn start`
2. Go to Stack Management -> Index Management -> Data Streams
3. Switch on the "Include stats" toggle
4. Verify that the "Storage size" column is shown in the table.
<!---
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
-->
## Summary
Fixes https://github.com/elastic/kibana/issues/165895
This PR limits which index settings are displayed on the index details
page, "Settings" tab in the edit mode. On serverless only a handful of
index settings will be editable by the user. The UI only prevents
displaying some index settings, but it's still possible for the user to
type in a setting that can't be edited. That is the case on dedicated as
well.
### How to test
1. Start Serverless ES and Kibana
2. Navigate to Index Management and create a test index
3. Click on the index name and on the details page click the tab
"Settings"
4. Toggle the "Edit mode" switch and verify that only editable settings
are displayed.
#### Screenshot
<img width="527" alt="Screenshot 2023-10-16 at 20 25 49"
src="e6678cca-3494-4c63-ae66-ace9c823d12d">
## Summary
Closes https://github.com/elastic/ingest-dev/issues/2396
Added a new kibana task that publishes Agent metrics every minute to
data streams installed by fleet_server package.
Opened the pr for review, there are a few things to finalize, but the
core logic won't change much.
To test locally:
- Install fleet_server package 1.4.0 from
[this](https://github.com/elastic/integrations/pull/8145) pr to get the
mappings
- Start kibana locally, wait for a few minutes for the metrics task to
run (every minute)
- Go to discover, `metrics-*` index pattern, filter on
`data_stream.dataset: fleet_server.*`
- Expect data to be populated in `fleet_server.agent_status` and
`fleet_server.agent_versions` datasets.
<img width="1787" alt="image"
src="615af9df-fe4b-4c17-8c8c-88646c403a18">
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
fix https://github.com/elastic/kibana/issues/166233 (this is a short
term fix that removes "analytics" references from the empty state page
in serverless observability analytics apps)
## Summary
This PR implements a standard way to have different static settings for
the serverless and ess (stateful) environments. It centralizes flags,
which were set using different approaches previously, in a single
configuration.
This aims to make it easier for developers to enable/disable parts of
the application in serverless projects.
Default:
```
sideNavEnabled: true,
ILMEnabled: true,
ESQLEnabled: true,
```
Serverless:
```
xpack.securitySolution.offeringSettings: {
sideNavEnabled: false, # Internal security side navigation disabled, the serverless global chrome navigation is used instead
ILMEnabled: false, # Index Lifecycle Management (ILM) functionalities disabled, not supported by serverless Elasticsearch
ESQLEnabled: false, # ES|QL disabled, not supported by serverless Elasticsearch
}
```
### Consume the settings
#### Server
- Plugin parsed `ConfigType`:
`this.config.settings.ESQLEnabled`
#### UI
- Plugin attribute:
`this.configSettings.ESQLEnabled`.
- Components can access it from Kibana services:
`useKibana().services.configSettings.ESQLEnabled;`
---------
Co-authored-by: Vitalii Dmyterko <92328789+vitaliidm@users.noreply.github.com>
