## Summary
Closes https://github.com/elastic/kibana/issues/184025
This PR enables the migration from Ace to Monaco in Dev Tools Console by
default in the main branch. All serverless projects will still have the
migration disabled by default. After 8.15 is branched, the migration
will be disabled there as well. The intended release version for this
migration is 8.16.
### Functional tests
This PR creates a copy of functional tests for Monaco Console and keeps
the tests for Ace in a separate folder. When the migration is released,
we can remove the code for Ace together with tests.
The Monaco tests are not the exact copy of the Ace tests, since some
functionality and autocomplete behaviour is slightly different in the
migrated Console. For example, the auto-closing of brackets works in
Monaco when typing something, but is not kicking in in the tests.
Flaky test runner
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Introducing the `search_homepage` plugin along with integration into
`enterprise_search` and `serverless_search` behind a feature flag. This
will allow implementing the feature gated behind the feature flag.
To test these changes you can enable the feature flag with the Kibana
Dev Console using the following command:
```
POST kbn:/internal/kibana/settings/searchHomepage:homepageEnabled
{"value": true}
```
You can then disable the feature flag with the following command:
```
DELETE kbn:/internal/kibana/settings/searchHomepage:homepageEnabled
```
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fix https://github.com/elastic/kibana/issues/185042
- Add a new `elasticsearch.maxResponseSize` config option
- Set this value to `100mb` on our serverless configuration file
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Description
In this PR, we implemented a view for managing inference endpoints. The
changes include the following items for both **Serverless** and
**Stack**.
- A blank page will be displayed if no inference endpoints are
available.
- A page displaying a list of inference endpoints. The user can view
various details about each endpoint, such as the endpoint itself, the
provider, and the type. The table supports pagination and sorting.
- Users can add a new inference endpoint using Elasticsearch models and
third-party APIs, including Hugging Face, Cohere, and OpenAI.
To keep the changes in this PR manageable, the following items are **out
of scope** but will be added in subsequent PRs
- Option to delete an inference endpoint
- Filtering and Search bar
- Information about allocations, thread.
- Icons for **Provider**
- Deployment status of underlying trained models
## Empty page in Stack Management
e2064ee8-3623-457f-8a04-19603e97e815
## Page with all inference endpoints in Stack Management
89bec450-1569-4425-b013-5058b577b95a
## Inference Endpoints Management in Serverless
bd8b6b71-0e09-49f4-aa9a-19338a1da225
---------
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR extends the features plugin to accept feature definition
overrides via Kibana configuration. The functionality is limited to the
Serverless offering only. Additionally, the PR updates Kibana serverless
configurations to include overrides based on the "simplified feature
toggles" proposals discussed with the solution teams.
The configuration might look like this:
```yaml
## Fine-tune the feature privileges.
xpack.features.overrides:
dashboard:
privileges:
### Dashboard's `All` feature privilege should implicitly
### grant `All` access to Maps and Visualize features.
all.composedOf:
- feature: "maps"
privileges: [ "all" ]
- feature: "visualize"
privileges: [ "all" ]
### All Dashboard sub-feature privileges should be hidden:
### reporting capabilities will be granted via dedicated
### Reporting feature and short URL sub-feature privilege
### should be granted for both `All` and `Read`.
subFeatures.privileges:
download_csv_report.disabled: true
url_create:
disabled: true
includeIn: "read"
### Maps feature is disabled since it's automatically granted by Dashboard feature.
maps.disabled: true
```
## How to test
Log in as the `admin` using SAML and navigate to the `Custom roles`
management section to edit role and see tuned role management UX:
<p align="center">
<img
src="ad6e4b07-53bd-4f5a-ae91-66d6534c711a"
/>
<img
src="8ab4d5a3-f719-42d5-a278-3aee87603c33"
/>
</p>
### Search project
```bash
yarn es serverless --projectType=es --ssl -E xpack.security.authc.native_roles.enabled=true
yarn start --serverless=es --ssl --xpack.security.roleManagementEnabled=true
```
Refer to the proposal document, `config/serverless.yml`, and
`config/serverless.es.yml` in this PR to see the specific changes made
for your project type:
Create a custom `custom-search` role and re-login as the user with this
role to test your project type (you need to manually type role name if
the role selector):
<p align="center">
<img
src="5088320b-3cc8-4de9-984c-d70fc6277659"
/>
</p>
### Observability project
```bash
yarn es serverless --projectType=oblt --ssl -E xpack.security.authc.native_roles.enabled=true
yarn start --serverless=oblt --ssl --xpack.security.roleManagementEnabled=true
```
Refer to the proposal document, `config/serverless.yml`, and
`config/serverless.oblt.yml` in this PR to see the specific changes made
for your project type:
Create a custom `custom-o11y` role and re-login as the user with this
role to test your project type (you need to manually type role name if
the role selector):
<p align="center">
<img
src="110572b1-f08a-4427-a687-5c2e0240a36b"
/>
</p>
### Security project
```bash
yarn es serverless --projectType=security --ssl -E xpack.security.authc.native_roles.enabled=true
yarn start --serverless=security --ssl --xpack.security.roleManagementEnabled=true
```
Refer to the proposal document, `config/serverless.yml`, and
`config/serverless.security.yml` in this PR to see the specific changes
made for your project type:
Create a custom `custom-security` role and re-login as the user with
this role to test your project type (you need to manually type role name
if the role selector):
<p align="center">
<img
src="2bec6ae2-8d19-4142-a479-9a81bc1fca14"
/>
</p>
__Fixes: https://github.com/elastic/kibana/issues/178963__
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Part of https://github.com/elastic/kibana/issues/181938.
Disables bfetch in serverless by overriding the `bfetch:disable`
advanced setting in the serverless.yml.
Bfetch was introduced to bypass the browser connection limit when
multiple search requests are made. Here's a picture of what the network
tab looks like with bfetch turned off in http1:
As the number of requests reaches the connection limit (6 on most modern
browsers), requests stall until a connection is available.
In Cloud/serverless, we have a http2 proxy. Http2 does not have this
browser connection limit. Turning off bfetch allows us to collect
metrics to decide if we want to remove it altogether:
| Bfetch enabled | Bfetch disabled (this PR) |
| --------------- | ------------------------- |
|
|
|
In my tests, the overall time for a sample dashboard to load was
actually smaller with bfetch disabled on http2 vs. bfetch enabled.
### Checklist
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
Added CSP and Cloud Defend plugins into Exclude Package list for oblt
projects in Serverless
Co-authored-by: Maxim Kholod <maxim.kholod@elastic.co>
Fixes https://github.com/elastic/kibana/issues/183166
## Summary
This small PR hides the previously mentioned setting in serverless and
adds a uiOverride to keep it always disabled too.
##### How to test
* Verify that the setting is disabled and hidden for serverless
* Verify that the setting is still functional for onprem
---------
Co-authored-by: Yulia Čech <6585477+yuliacech@users.noreply.github.com>
## Summary
This adds a new config item for the `search_notebooks` catalog to allow
fetching it from a URL instead of only serving the static notebooks
packaged with Kibana. This will be used in ES3 to ensure we have the
latest notebooks, but also give us the ability to change which notebooks
are served without making kibana code changes.
## Summary
There has been a consistent failure in a Discover-related test set in
the kibana-ES-serverless verification job, meaning that ES-Kibana
compatibility has drifted.
Error details:
```
+ "Encountered an unknown error: status_exception
+ Root causes:
+ status_exception: Parameter validation failed for [/_search]: The http parameter [max_concurrent_shard_requests] (with value [5]) is not permitted when running in serverless mode"
+ "Encountered an error with the number of CSV rows generated fromthe search: expected rows were indeterminable, received 0."
at Context.<anonymous> (reporting.ts:182:33)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at Object.apply (wrap_function.js:73:16)
```
This tracked back to a feature added for reporting awhile back, which
created a config schema field for the `max_concurrent_shard_requests`
parameter in the search queries:
https://github.com/elastic/kibana/pull/170344/files#diff-7bceb37eef3761e1161cf04f41668dd9195bfac9fea36e734a230b5ed878a974
Most of the changes in this PR are in test code. I created "Test" which
extend protected methods in the original classes. This was done to
remove the `@ts-expect-errors` lines of code.
## Summary
This PR links the ESQL functionality in security solution to the
`discover:enableESQL` advanced setting. The advanced setting will only
be present in ESS, but not serverless
The way this should work to maintain parity with the rest of Kibana such
as discover and stack rules:
- By default ES|QL will be enabled across all Kibana
- When the ES|QL advanced setting is disabled:
- Timeline
- ES|QL tab should not be accessible on any newly created timelines
- Existing Timelines with an ES|QL query should still have the tab
accessible
- Rules
- New ES|QL rule should not be available to be created in the *Rule
Creation* workflow
- Existing ES|QL rules should still run and be able to be edited
**Timeline Demo Video:**
d5429be9-de37-43e2-882d-687b3371beb4
**Rules Demo Video:**
7df2fd11-bd2b-4e50-ad97-b6e1d0f7867a
---------
Co-authored-by: Vitalii Dmyterko <92328789+vitaliidm@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Closes https://github.com/elastic/kibana/issues/181555
## Summary
This PR disables the `_source` field in the Mappings editor's advanced
options when in serverless.
**How to test:**
1. Start a serverless project.
2. Go to Index Management and start creating an index template or a
component template.
3. In the Mappings step, go to the "Advanced options" tab.
4. Verify that the `_source` field is not displayed and that creating a
template doesn't add this property to the Es request.
5. Verify that, in stateful Kibana, the `_source` field still exists and
works as expected.
<!--
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
-->
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Renames the setting from discover:enableESQL to enableESQL and moves it
to the General Section
<img width="985" alt="image"
src="ebafe7cd-4695-413c-a0e3-90e3574139e5">
We checked the telemetry and a very small percentage of users have
switched off this setting. It was also on tech preview so we decided in
favor of this change as more and more applications are using this
setting to hide ESQL from their applications.
<img width="850" alt="image"
src="404f2494-8a17-4a9d-943a-d9d72b324bc0">
### Release note
We renamed the advanced setting which hides the ES|QL from the UI from
`discover:enableESQL` to `enableESQL`. You have to switch it off again
if you want to disable ES|QL features from your kibana applications.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fixes https://github.com/elastic/kibana/issues/167955
To test:
```
# Build Kibana
node scripts/build --skip-os-packages --skip-cdn-assets --skip-canvas-shareable-runtime --skip-docker-ubi --skip-docker-ubuntu --skip-docker-fips --skip-node-download
# Run Kibana
cd build/default/kibana-8.15.0-SNAPSHOT-darwin-aarch64
./bin/kibana
# In a different terminal find the pid and send the signal
ps -e | grep bin/node
kill -s SIGUSR2 PID # use pid from ^
# verify heap snapshot is created
ls ./data
# expect to see a file like
Heap.20240423.132053.37884.0.001.heapsnapshot
```
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Resolves https://github.com/elastic/kibana-team/issues/827.
`1m` wasn't sufficient for some EQL queries in serverless. After talking
with @peluja1012 @marshallmain and @yctercero, it was decided the
default of `5m` would be a better starting point for all serverless
projects. Tradeoffs have been communicated in
https://github.com/elastic/kibana-team/issues/827 and offline, it was
decided it's still worth increasing the timeout for EQL rules for now.
Increasing the metric reset interval for the `/api/task_manager/metrics`
API to 2 minutes from 30 seconds. We believe 30 seconds is too
aggressive based on data from the overview cluster that shows metrics
being collected anywhere from 10 seconds (the expected interval) to 2
minutes.
Updates the text on the landing page
**Before**
**After**
Adds a new feature flag to allow the `Create filebeat configuration`
card to be hidden. This has been
[requested](https://github.com/elastic/enterprise-search-team/issues/7075)
for the serverless search project and so the `serverless.es.yml` has
been updated to include this flag.
**Before**
**After**
## Summary
This PR enables the `xpack.ml.nlp.enabled` flag for the security project
in support of enabling ELSER for use with the Elastic Assistant.
Reference issue: https://github.com/elastic/security-team/issues/7995
Corresponding `project-controller` PR's here:
https://github.com/elastic/project-controller/pull/674https://github.com/elastic/project-controller/pull/751
---
As discussed in the above reference issue, this PR adds a task for
cleaning up NLP models at a regular interval if the `productTier` is not
`complete`. This `serverless-security:nlp-cleanup-task` will run every
`6hr`, attempting to cleanup any `pytorch` NLP models/aliases that were
either leftover from a `productTier` downgrade, or were manually created
using the ES trained model API's.
Note I: As recommended by the ResponseOps folks, the task is _always
registered_ regardless of the current `productTier`. When executed on
the `complete` `productTier`, an `UnrecoverableError` is thrown, which
will remove the task and prevent it from running again until Kibana is
restarted.
Note II: While the task interval is every `6hr`, when initially
registered we manually kick off an execution to ensure that any
resources are immediately cleaned up when a `productTier` change is
performed.
---
Both `complete` and `essentials` API integration tests have been added
to cover the above behavior. Please see:
*
[.../test_suites/genai/nlp_cleanup_task/basic_license_essentials_tier/task_execution.ts
](https://github.com/elastic/kibana/pull/175358/files#diff-b0cae16c5c6f77a90d53fcfab3bda6400ec13fe7f200d8c1f0dd3f1a0109af9f)
*
[.../test_suites/genai/nlp_cleanup_task/trial_license_complete_tier/task_execution.ts](https://github.com/elastic/kibana/pull/175358/files#diff-c69ec08309855f236ab90d4ba73f40636e52d706679bbab530956ff8d5560097)
---
Additionally, Event Based Telemetry (EBT) was added so cleanup, and any
associated failures can be tracked. Please see the
`NLP_CLEANUP_TASK_EVENT`, which has the below schema:
```
failedToDeleteCount: number; // Total number of NLP models failed to be cleaned up
message: string; // General message from task completion, either summary of task or error message
productTier: string; // Current productTier when task was run
totalInstalledCount: number; // Total number of NLP models deployed when task started
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
### Summary
This PR enables the NLP models management page in the ML plugin for
Observability serverless projects and updates how the Obs AI Assistant
checks if ELSER is ready to be used.
## Summary
issue: https://github.com/elastic/kibana/issues/179162
This PR brings the new navigation, with the solution-centric IA
(Information Architecture), to the ESS (stateful) Security Solution.
To do so, the implementation of the `navigationTree$`, which was
previously only implemented in serverless, has been integrated inside
the generic "security_solution" plugin, so now it is available for ESS
and serverless offerings.
In ESS users can still choose the navigation version, so we have to
temporarily keep supporting both, the classic and new navigation
implementations. After the rollout, the classic navigation components
will be removed and the unified links architecture should be reassessed.
The issue for the cleaning:
https://github.com/elastic/kibana/issues/179572
### Rollout
The new solutions navigation will not be available for customers on
8.14, it will only be enabled for internal Elastic users (via
Lauchdarkly), who will have the ability to opt out of it from their
profile menu. We'll collect feedback and telemetry and address any bugs
or improvements (together with the Kibana platform team). The plan is to
start making it available to customers in 8.15.
### Testing
Unless we add the _kibana.yml_ configurations to enable the new
navigation, the regular classic navigation will be displayed, it should
keep working the same way without any change. The new landing pages
(`Assets`, `Investigations`, `Machine Learning`,...), that exist only
when using the new navigation, should not be accessible using the
classic version.
To enable the new navigation add the following _kibana.yml_ configs:
```
xpack.cloud_integrations.experiments.enabled: true
xpack.cloud_integrations.experiments.flag_overrides:
"navigation.solutionNavEnabled": true
xpack.cloud.id: "ftr_fake_cloud_id:aGVsbG8uY29tOjQ0MyRFUzEyM2FiYyRrYm4xMjNhYmM="
xpack.cloud.base_url: "https://cloud.elastic.co"
xpack.cloud.deployment_url: "/deployments/deploymentId"
```
And enable the advanced setting
### Screenshots
The app switcher:
<img width="293" alt="app switcher"
src="0a638b8f-fdc0-4d1a-b8d3-607e487215f4">
---
New Assets landing page:
<img width="549" alt="assets landing"
src="17bc8a94-02b4-4996-b9f5-8731ba81ac43">
---
For `Stack Management` we set the nav panel flyout and the cards landing
page, this is temporary until a decision on how to show Stack Management
links is made:
<img width="954" alt="stack management"
src="27ce6534-0508-4804-b224-8dc409042825">
---
The switch to go back to the classic nav is in the profile menu at the
top-right corner:
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Søren Louv-Jansen <soren.louv@elastic.co>
Co-authored-by: Vitalii Dmyterko <92328789+vitaliidm@users.noreply.github.com>
Co-authored-by: jennypavlova <dzheni.pavlova@elastic.co>
Co-authored-by: Katerina <aikaterini.patticha@elastic.co>
Co-authored-by: Sébastien Loix <sebastien.loix@elastic.co>
Co-authored-by: Kurt <kc13greiner@users.noreply.github.com>
Co-authored-by: Justin Kambic <jk@elastic.co>
Co-authored-by: Julia Bardi <90178898+juliaElastic@users.noreply.github.com>
Co-authored-by: Paul Tavares <56442535+paul-tavares@users.noreply.github.com>
Co-authored-by: Nathan Reese <reese.nathan@elastic.co>
Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com>
Co-authored-by: Milton Hultgren <milton.hultgren@elastic.co>
Adds a setting in the `aiAssistantManagementSelection` plugin that
allows users to set the preferred Assistant type. The allowed values
are:
- `default`: the status quo, which is: show the Observability AI
Assistant in Observability apps, the Security AI Assistant in Security
solution apps, and none in the other apps.
- `never`: Never show any AI Assistant.
- `observability`: Show the Observability AI Assistant everywhere,
except for Security solution apps.
- `security`: ~Show the Security AI Assistant everywhere, except for
Observability apps.~
it is up to the solutions to respect the setting. See
[x-pack/plugins/observability_solution/observability_ai_assistant_app/public/hooks/is_nav_control_visible.tsx](https://github.com/elastic/kibana/pull/179233/files#diff-753a9aae298da45c5bd96bc2cacaa437ac53e165069ad6323973dd3c2879d4ae)
for an example. It should also be set in the specific Serverless config:
[config/serverless.oblt.yml](https://github.com/elastic/kibana/pull/179233/files#diff-358680d2b72268283dca4f3efb976cd924b7b5bce822e07c8871d061b4bb339e).
Update: this is now labeled as an Observability-only setting.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Disable the Obs AI Assistant plugin in Serverless Search & Serverless
Security. Some work was done to move `observabilityAIAssistant` from
`requiredPlugins` to `optionalPlugins` in upstream dependencies.
Closes https://github.com/elastic/kibana/issues/166999
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fix https://github.com/elastic/kibana/issues/178180
- Add a new `core.lifecycle.disablePreboot` (internal) config option to
forcefully disable Core's `preboot` phase.
- Enable the option in the serverless configuration file
Gain is around 150/200ms on local developer machine, which translates to
~300/500ms on serverless environment
Fixes https://github.com/elastic/kibana/issues/176420
## 🍒 Summary
This PR copies the SLO code that was inside the Observability app into
its own app under `observability-solution/slo` folder.
4f6b8dfb-9612-4d30-ad50-4ee5c55a9c32
## ✔️ Acceptance criteria
- URL of new app: `app/slos`
- Design and functionality are not changed.
- Git history has been retained for all files in
`x-pack/plugins/observability_solution/slo`.
- SLO should appear on server less
- SLO code inside `observability_solution/observability` code has been
removed. A new clean up round might be needed though for possible
leftovers.
- Burn rate rule is registered within the new slo app
- SLO embeddables are moved inside the new slo app
- overview
- alerts embeddable
- error budget burn down
- Alerts table configuration registration for slo details page and
alerts table embeddable is still done in the observability app. Response
Ops team is working on removing the need to register the alert table
anyway
- Slo app is wrapped into `ApplicationUsageTrackingProvider` which will
send slo `Application usage` information tracked by the `slo` appId
- Redirect old `app/observability/slos` route to `app/slos`
- Rename old `xpack.observability.slo` keys to `xpack.slo` in the
translation files
## 🌮 How to test
Design and functionality didn't change, so simply navigate to existing
slo pages and try to break it
- Slo list page
- group by
- unified search
- toggle buttons
- actions
- Slo creation
- try group by as well
- Slo detail page
- Actions on top
- navigate to overview and alerts tabs
- Create SLO flyout in Logs Explorer
- Create burn rate rules and verify they appear on rules page
- Verify SLO alerts appear on Alerts page and slo details page
- Embeddables
- Through the dashboard app
- Using the attach to dashboard action on the slo card item on slo list
page and the error budget burn down chart on the slo detail page
- SLOs only for platinum users
- Permissions
- Spaces
## TODO
- [x] Move slo stuff from observability folder to new slo plugin
- [x] Remove old slo stuff from observability folder
- [x] Update references
- [x] Fix typescript and eslint errors
- [x] Paths
- [x] Locators
- [x] Burn rate rule registration
- [x] Embeddable Alerts table configuration registration
- [x] Embeddables
- [x] Translations
- [x] Verify plugin.ts files contain all registration logic
- [x] public
- [x] server
- [x] Final cleanup for observability folder
- [x] Run tests
- [x] Application Usage (Telemetry)
- [x] Permissions
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: shahzad31 <shahzad31comp@gmail.com>
Co-authored-by: Coen Warmer <coen.warmer@gmail.com>
Towards: #176585
This PR removes the task skipping logic from TaskManager, PRs for
Alerting and Actions will follow.
## To verify
Rules and actions should be still working as expected.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
At present, we have the MKI Kibana Controller disable Synthetics in
production Serverless by default.
This patch will make Synthetics enabled by default, so at the point when
we want to remove the injected disable config on the controller side,
Synthetics will be available to all production Observability projects
immediately.
## Summary
Closes https://github.com/elastic/kibana/issues/174771
While the security team works on Custom Roles for serverless, we want to
hide the content behind a feature flag.
An existing config option that was used to hide the Roles UI during the
initial phases of serverless has been repurposed, and will now toggle
both the Roles UI and the Roles Routes
`xpack.security.confg.ui.roleManagementEnabled` has been changed to
`xpack.security.confg.roleManagementEnabled` and will have to be set to
`true` in a config file while in serverless mode to show the Roles card
on the management screen and enable the UI/routes.
## Reviewers
Ive included a `viewer`:`changeme` user for testing (It will be removed
after approval).
## Testing
### xpack.security.config.roleManagementEnabled
1. In your `kibana.yml`, add
`xpack.security.confg.roleManagementEnabled: true`
2. Start up in serverless mode locally, login in with
`elastic_serverless`:`changeme`
3. Click `Project Settings` > `Management`
4. `Roles` card should display under `Other`
5. Navigate to `Roles`, it displays, but the `Edit Roles` page does not
work yet.
### Test as Viewer
1. In your `kibana.yml`, add either above option as you prefer
2. Start up in serverless mode locally, login in with
`viewer`:`changeme`
3. Click `Project Settings` > `Management`
4. `Roles` card should NOT display under `Other` and the `roles` URL
should not work.
## Screenshots
Roles card
<img width="1281" alt="Screenshot 2024-02-05 at 3 22 12 PM"
src="a1285ada-7ff7-495f-88a6-9847b3245518">
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>
## Summary
Part of https://github.com/elastic/kibana/issues/144276
- Introduce the concept of browser-side logging configuration, via a
`logging.browser` config prefix
- Allow to configure the log level for the root browser logger via
`logging.browser.root.level`
- Set the default level to `info` for both dev and production mode
(consistent with server-side logging)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Resolves: https://github.com/elastic/kibana-team/issues/721
As we discovered by an issue, it's an anti pattern to timeout-and-retry
requests part of the polling cycle because it's basically putting the
requests back in the same queue that it once was ahead on.
Therefore we would like to make TaskManager's update-by-query request
timeout configurable.
Closes https://github.com/elastic/kibana/issues/165672
### Summary
This PR adds support for custom output and Fleet server hosts with
proxies in serverless mode:
- Proxies are re-enabled in serverless.
- It is possible to add custom Fleet server hosts in serverless, with
the constraint that the host URL must match the Elasticsearch URL of the
default host.
- New Elasticsearch outputs must also have the default host URL.
### Testing
The below requirements should be tested in serverless mode for
observability and security project types:
```sh
# elasticsearch
yarn es serverless --kill
# kibana: one of
yarn serverless-oblt
yarn serverless-security
```
⚠️ In addition, stateful mode should **not** be affected by any of these
changes.
#### Config
In order to test this change, you will need the following configuration
to mirror a serverless setup.
Create a `serverless.dev.yml` if you don't have one already and set a
project id (this is required for Fleet's `cloud.isServerlessEnabled` to
correctly be `true`):
```yaml
xpack.cloud.serverless.project_id: test-123
```
In `kibana.devl.yml`, make sure the default Fleet server hosts and
default output have the expected ids:
```yaml
xpack.fleet.fleetServerHosts:
- id: default-fleet-server
name: Default Fleet server
is_default: true
host_urls: ['https://host.docker.internal:8220']
xpack.fleet.outputs:
- id: es-default-output
name: Default output
type: elasticsearch
is_default: true
is_default_monitoring: true
hosts: ['https://host.docker.internal:9200']
```
#### Requirements
- [ ] User can create proxy configurations in the Fleet UI and API.
- [ ] User can create new Fleet server host via the UI
- The Fleet Server Hosts dropdown is disabled and set to the default
host URL
- [ ] User can create new Fleet server host via the API
- The request should succeed if the host URL is set the the default one
- Otherwise the request should fail
- [ ] User can add a proxy config to the Fleet server host config
- [ ] User can select a custom Fleet Server host configuration from the
Agent policy settings page
- [ ] User can create new Fleet Elasticsearch output via the UI
- The Hosts dropdown is disabled and set to the default Elasticsearch
URL
- [ ] User can create new Fleet Elasticsearch output via the API
- The request should succeed if the host URL is set the the default one
- Otherwise the request should fail
- [ ] User can add a proxy config to the Elasticsearch output
- [ ] User can select a custom Elasticsearch output configuration from
the Agent policy settings page
- [ ] User can create a custom Logstash output with proxy
- [ ] User can create a custom Kafka output with proxy
### Checklist
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### Screenshots
Fleet settings (proxies available):
Adding and editing a Fleet server host:
Adding and editing an Elasticsearch output:
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Introduced the console.ui.embeddedEnabled (boolean) value to allow
disabling the embedded console in serverless security and observability
projects.
This also replaces an index management specific config value to do that
same thing which is removed here as well. Adding this console
configuration will allow us to have one value to set instead of N which
was the pattern I had started to use, but then decided against.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
Closes https://github.com/elastic/kibana/issues/172922
## Summary
This PR:
- Integrates the new Settings application
(`packages/kbn-management/settings/application`) into stateful Kibana
and removes the old `management_app` from the
`src/plugins/advanced_settings` plugin.
- Adds support for section registry in the new Settings application, so
that other plugins can add their own sections to the Advanced settings
app.
- Adds functionality for disabling saving of settings based on the
provided capabilities of the current user.
<img width="1352" alt="Screenshot 2024-01-23 at 16 46 03"
src="1f3b7088-58e2-46e8-a7dd-ae0fc346b4ba">
<br><br>
"Usage collection" section in Global settings:
<img width="1099" alt="Screenshot 2024-01-23 at 16 48 24"
src="ebc54ad5-348b-46dd-a047-b418ddc7ba4f">
### How to test
**Testing Advanced settings in stateful Kibana:**
1. Start Es with `yarn es snapshot` and Kibana with `yarn start`
2. Go to Stack Management -> Advanced Settings
3. Verify that the app functions correctly. Both tabs (for space and
global settings) should be displayed, setting fields should be editable
and saveable, etc.
**Testing the section registry**
Currently, `telemetry_management_section` is the only plugin that
registers a section - the "Usage collection" section under the "Global
settings" tab. This should work correctly in stateful Kibana.
1. Start Es with `yarn es snapshot --license=trial` and Kibana with
`yarn start`
2. Go to Stack Management -> Advanced Settings and select the "Global
settings" tab
3. Scroll down and verify that the "Usage collection" section is
displayed and works as expected.
**Testing with different capabilities:**
1. Start Es with `yarn es snapshot` and Kibana with `yarn start`
2. Go to Stack Management -> Roles
3. Create a role that has "Read" access to Advanced settings and one
that doesn't have any access.
4. Create users with each of these two roles.
5. Log in with these users and verify that the user with "Read" access
can see the app but cannot edit it, and the user with no privileges
cannot access the app.
**Testing Advanced settings in serverless Kibana:**
The Advanced settings app in serverless shouldn't be affected by these
changes.
1. Start Es with `yarn es serverless` and Kibana with `yarn
serverless-{es/oblt/security}`
2. Go to Management -> Advanced Settings
3. Verify that the app functions correctly. There shouldn't be any tabs
as there are no spaces.
<!--
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
-->
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
