Depends on https://github.com/elastic/kibana/pull/221610
This PR adds the import index workflow to privileged user monitoring and
API changes required to support it.
### API Enhancements
* **New API for privilege monitoring index creation**: Added a new API
endpoint (`PUT /api/entity_analytics/monitoring/privileges/indices`) to
create indices for privilege monitoring with support for `standard` and
`lookup` modes. This includes the implementation of request and response
schemas (`create_indidex.gen.ts`, `create_indidex.schema.yaml`).
[[1]](diffhunk://#diff-68329bb90dea945f343e1637990d5d05bc159e0aa2511ef1e45d37ed1a6cda51R1-R41)
[[2]](diffhunk://#diff-e979499654a27b3c1930d63c5b1002113c1c3f53f84ce27a4d75a5c492717a96R1-R42)
* **Updated privilege monitoring health response**: Modified the health
response schema to include a `status` field and an optional `error`
object for detailed error handling
(`privilege_monitoring/health.gen.ts`,
`privilege_monitoring/health.schema.yaml`).
[[1]](diffhunk://#diff-00f39a3e65a336eaddf7d3203d1370d910f5ecd2062b6cc21d9c06922c12884eR19-R28)
[[2]](diffhunk://#diff-83afa72b7a1fc48f3cc063e9fb855190d3525228bc0488fb8b871e112b90e961L22-R33)
### Frontend Integration
* **Introduce the create index modal that opens when the create index
button is clicked.
* **Onboarding modal improvements**: Updated the `AddDataSourcePanel`
component to handle index creation more robustly by passing callbacks to
the modal (`add_data_source.tsx`).
* **Error handling in UI**: Enhanced the `PrivilegedUserMonitoring`
component to display error callouts when privilege monitoring data fails
to load (`privileged_user_monitoring/index.tsx`).
[[1]](diffhunk://#diff-273ad32c97dcf15c6c6054fd7c5516d587132674578d25986b235cd174c75789R22-R26)
[[2]](diffhunk://#diff-273ad32c97dcf15c6c6054fd7c5516d587132674578d25986b235cd174c75789R38-R51)
### How to test it?
* Go to the priv mon page with an empty cluster
* Click on the data source by the index button
* Search for available indices, it should return indices with
`user.name.keyword` fields
* Click 'create index' and create a new index
* Choose the created index and click 'Add privileged users'
* You should be redirected to the dashboard (The API is currently not
working)
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.
Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.
- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
- Adds configuration for the product intercept in `oblt`, `es` and
`security` serverless offerings, alongsides stateful offering too. The
configuration provided sets the intercept to display every 90days, this
is configurable through the config `xpack.product_intercept.interval`.
The intercept can also be turned off through the config
`xpack.product_intercept.enabled`
- Also tweaks prompter timer implementation to accommodate inherent
[issue with long timer
delays](https://developer.mozilla.org/en-US/docs/Web/API/Window/setTimeout#maximum_delay_value)
in the browser
- Adjusts the signature of the `registerIntercept` method, such that a
deferred value to be evaluated when the intercept is to be displayed is
passed. This unlocks the ability to have consumers provide dynamically
imported modules that provide the config for the intercept, see
0e07892217
for an example.
### How to test
- Add the following config to your `kibana.dev.yml` file;
```yml
xpack.product_intercept.enabled: true
# we set the interval to 30s so the wait long period to display the
intercept is bearable
xpack.product_intercept.interval: '30s'
```
- Start kibana in stateful, and serverless mode, in either scenario you
should be presented the product intercept, with the intercept
specifically stating the current product the user is interacting with.
See below for an example of observability solution;
https://github.com/user-attachments/assets/6ca6baf2-58d3-4002-ac94-ec6e9a0902ae
<!--
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.
Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.
- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...
-->
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Closes https://github.com/elastic/kibana/issues/179173.
## Rule type
A dedicated stack rule type was created `datasetQuality.degradedDocs`.
<img width="1759" alt="image"
src="https://github.com/user-attachments/assets/5004a08d-6f12-4f5e-b27f-5f4db242dcf0"
/>
<img width="2318" alt="image"
src="https://github.com/user-attachments/assets/f8b2664a-f1c6-48c5-a617-c6f1b79bf0f7"
/>
This new rule is aggregated by default using `_index` and could be
further configured by the user (e.g. user can also aggregate by
`cloud.provider`).
A new rule type was needed to be created since there is no actual way to
aggregate all documents in a dataStream if we use a DataView like
`logs-*-*`. Inside datasStream documents there is no indication about
the dataStream where they belong to, instead we just have `_index` which
contains backingIndexName instead of actual index.
It's important to note, that this rule type is also visible from
`Observability > Alerts`, which is useful specially for serverless.
https://github.com/user-attachments/assets/000aee51-4895-4f4c-9484-924ace4325c5
## Role Based Access-Control (RBAC)
RBAC for dataset quality alerts is defined within dataQuality kibana
feature.
We have three privileges defined:
1. `all`: This privilege now contains a subFeature `manage_rules` that
will allow for more granularity on alerting level. It's by default
assigned to `all` but can be disabled.
2. `read`: This privilege is only related to serverless (when we don't
have yet custom roles).
https://github.com/user-attachments/assets/70ed5bde-bf45-4024-b448-228799fcaf71
3. `none`: This privilege is only relevant for stateful (in serverless
we don't have custom roles).
## 🎥 Demo
### Serverless
#### `all` privileges
https://github.com/user-attachments/assets/8dad6e30-a261-4a69-979f-6dfc2a41c888
#### `read` privileges
https://github.com/user-attachments/assets/e1cb108d-22a0-4e7f-b252-9cc12d1e9d65
### Stateful
#### `all` privileges
https://github.com/user-attachments/assets/d96f3b70-35b2-466b-aa59-a07190d24d93
#### `all` privileges with subFeature disabled
https://github.com/user-attachments/assets/808ab811-9320-43e4-b2a6-06d530a78b82
#### `none` privileges (Stateful)
https://github.com/user-attachments/assets/18f2a2d6-d825-4713-acea-0d72f451e9ab
## How to test?
1. run synthrace scenario `degraded_logs` in live mode
```
node scripts/synthtrace degraded_logs --live
```
2. Open dataset quality page (/app/management/data/data_quality)
3. Select `synth.3` dataset
(/app/management/data/data_quality/details?pageState=(dataStream:logs-synth.3-default)
4. Click on `Actions` and then select `Create rule`
5. Fill out the alert form
6. Go to `Observability > Alerts` or `Stack management > Alerts`
(/app/observability/alerts)
## Release note
Adds the Create alert rule action to dataset quality page and dataset
quality details. This allows you to generate an alert when the
percentage of degraded docs on the chart crosses a certain threshold.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@elastic.co>
Co-authored-by: Faisal Kanout <faisal.kanout@elastic.co>
# Overview
This pull request adds capabilities associated with the [Privileged
Access Detection (PAD) anomaly integration
package](https://www.elastic.co/docs/reference/integrations/pad) as a
first-class-citizen into the Entity Analytics Privileged User Monitoring
feature.
# How to test
- Pull this branch into your local machine
- Ensure that the security experimental flag
`privilegeMonitoringEnabled` has been enabled, for example by setting
`xpack.securitySolution.enableExperimental:
[privilegeMonitoringEnabled]` in your `config/kibana.dev.yml` file
- Start Elasticsearch and Kibana
- From the
[security-documents-generator](https://github.com/elastic/security-documents-generator)
repository, run the following command: `yarn start
privileged-user-monitoring` (ensuring your config is pointing to your
locally running Elastic cluster). This will load "source" events that
are anomalous in nature. You can run this command more than once if
desired to upload more than 10 users.
- Grab the CSV file that the generator created in its console output.
- Open the Entity analytics page and navigate to dashboards (by clicking
"Go to dashboards")
- Add the privileged users from the previous step to the privileged
users index using the CSV File Upload option.
> [!NOTE]
> Any errors regarding risk scoring are unrelated to this PR, and are
being resolved separately
- You should see a panel that says "Enable Privileged access detection".
Click "Install", and you'll meet a loading state.
- Once complete, you shouldn't see any results. That's because, even
though we **install** the ML jobs by default, we don't **run** them by
default.
- Click "ML Job Settings", and note that only `pad` jobs should be
displaying in this callout. Feel free to test this callout's links and
filtering options.
- Click "Run job" next to the job called
`pad_linux_high_count_privileged_process_events_by_user`. This is the
job for which we have anomaly data.
- Click away from the callout. **You still shouldn't see data.** That's
because there just aren't any anomalies "today".
- Change the global date filter at the top of the screen to "Last 30
days".
- You should see something similar to this:
<img width="1441" alt="Screenshot 2025-06-16 at 12 50 25 AM"
src="https://github.com/user-attachments/assets/2b3f11f2-f45d-4716-bb8e-79d2b585aa3e"
/>
- Congrats for making it this far! Some things to play around with:
- Click around on the anomaly filters (i.e., click 25-50), and notice
that the results will change.
- Ensure that the ordering of the users is based on the **highest single
anomaly score in any visible bucket**. Meaning, if user `samwise` has
only one anomaly, but its score is 99, and user `frodo` has dozens of
anomalies, but no higher than, say, 80, `samwise` will be on the top of
the list.
- Click the user names to open the appropriate user flyout
- Change the global time filter to change ranges, and ensure the data
shows up appropriately. The buckets will try to roughly show 30 buckets
total, but will have a maximum of 3 hours, meaning if your range is too
small, it may show fewer than 30 buckets. This is intentional behavior,
as the PAD jobs have an anomaly job window of 3 hours.
- Click the button that says "View all in Anomaly Explorer", which uses
the currently selected global time range, and compare results
> [!WARNING]
> Remember that the users in the privileged user monitoring table are
only those that you designated as privileged users in a previous step.
In contrast, the Anomaly Explorer page shows **all** users.
Additionally, note that there may be very slight differences between the
swimlanes, because ES|QL calculates the bucket dates slightly
differently than the Anomaly Explorer. **This should not affect the
results themselves, only the buckets that an individual anomaly score
might fall in.** You might see a single anomaly fall into one visual
bucket instead of another, but the date should be correct.
# Helpful hints
If you'd like an easy way to "reset" the Privileged Access Detection
package and delete its associated jobs (and anomaly data), so that you
can redo the onboarding flow, you can run the below commands (changing
any credentials as necessary):
```shell
curl "http://localhost:5601/api/fleet/epm/packages/pad/0.5.0" \
-X 'DELETE' \
-H 'elastic-api-version: 2023-10-31' \
-H 'kbn-xsrf:true' \
--user elastic:changeme
curl "http://localhost:5601/internal/ml/jobs/delete_jobs" \
-H 'kbn-xsrf:true' \
--user elastic:changeme \
-X 'POST' \
-H 'elastic-api-version: 1' \
-H 'x-elastic-internal-origin:kibana' \
-H 'Content-Type: application/json' \
--data-raw '{
"jobIds": [
"pad_linux_high_count_privileged_process_events_by_user",
"pad_linux_high_median_process_command_line_entropy_by_user",
"pad_linux_rare_process_executed_by_user",
"pad_okta_high_sum_concurrent_sessions_by_user",
"pad_okta_rare_host_name_by_user",
"pad_okta_rare_region_name_by_user",
"pad_okta_rare_source_ip_by_user",
"pad_okta_spike_in_group_application_assignment_changes",
"pad_okta_spike_in_group_lifecycle_changes",
"pad_okta_spike_in_group_membership_changes",
"pad_okta_spike_in_group_privilege_changes",
"pad_okta_spike_in_user_lifecycle_management_changes",
"pad_windows_high_count_group_management_events",
"pad_windows_high_count_special_logon_events",
"pad_windows_high_count_special_privilege_use_events",
"pad_windows_high_count_user_account_management_events",
"pad_windows_rare_device_by_user",
"pad_windows_rare_group_name_by_user",
"pad_windows_rare_privilege_assigned_to_user",
"pad_windows_rare_region_name_by_user",
"pad_windows_rare_source_ip_by_user"
],
"deleteUserAnnotations": true,
"deleteAlertingRules": false
}
'
```
# What's left?
- This PR does not implement "proactive" permissions checks. Instead,
relevant permission issues will cause requests/queries to fail, which
are appropriately shown in the UI for troubleshooting. This proactive
check will be tackled in [this (private) followup
issue](https://github.com/elastic/security-team/issues/12822).
- There is not yet a way to "upgrade" the package directly from the
Privileged User Monitoring screen. This would be a nice addition later
on, instead of requiring users to navigate to the integration page. This
behavior will be tackled in [this (private) followup
issue](https://github.com/elastic/security-team/issues/12823).
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Addresses https://github.com/elastic/kibana/issues/151636 .
This test was leftover from migration. Does not appear to be used.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This PR removes the current GET user route as it works with IDs and we
do not use IDs in privmon.
We'd rather remove this before it gets documented and pushed out. We can
revert this PR later if needed
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Mark Hopkin <mark.hopkin@elastic.co>
The PR skips test suites for ES `9.0.*` which were failing when these
tests ran in Kibana `8.19` branch against ES `9.0.*` in forward
compatibility runs. The tests were failing because the index privilege `read_failure_store` isn't available available in ES 9.0.
This PR closes#221508.
https://github.com/user-attachments/assets/0b57487a-7188-4722-99dc-5cb44c15f129
- Added a new API endpoint to edit the label of private locations.
- When a label is updated, all monitors deployed in that location are
automatically updated to reflect the change.
- The UI now allows users to edit only the label of a private location.
- Added comprehensive API tests to cover the new functionality.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
fixes https://github.com/elastic/kibana/issues/211540
User will be able to choose maintenance window in the form
<img width="1723" alt="image"
src="https://github.com/user-attachments/assets/c4d75aff-687f-40d3-a614-160e99ce9ac2"
/>
A callout will be displayed on the form
<img width="1728" alt="image"
src="https://github.com/user-attachments/assets/124727bd-0bb6-4934-9406-a36c3584670a"
/>
### Task manager
When changes are made to maintenance windows, those are sync via task
manager to private location monitors, public location monitors are
automatically synced as well in already existing task.
### Testing
Create a maintenance window in stack management UI, apply it to monitor,
make sure, it never runs during maintenance window.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR passes the logs essentials tier setting to the local ES test
server in FTR test configs. This is required by elasticsearch serverless
to disable certain features.
## Summary
This PR adds the ability to upload a CSV file with privileged users from
the Entity Analytics pages
## Changes
### Backend
- Added (or completed) the upload CSV route:
`/api/entity_analytics/monitoring/users/_csv`
- Added shared utilities for batching with Node streams
- Added bulk processing actions for the upload
- Parsing users from CSV
- Soft delete for omitted users
- Batch upsert via the bulk API
- Added a check for installing all required privmon resources
### Frontend
- File uploader components
- File validation logic
- Updated EA privmon page to account for the new flow
- Added managing users panels
- open upload flow (same as asset criticality)
## Screen recording
https://github.com/user-attachments/assets/7956f1cf-49e0-4430-8c23-7d6178a15342
## How to test
#### Prerequisite
Make sure you have a CSV file with usernames
Check
[here](https://gist.github.com/tiansivive/0be2f09e1bb380fdde6609a131e929ed)
for a little helper script
Create a few copies where some of the users are deleted, in order to
test soft delete
1. Start up kibana and ES
2. Navigate to Security > Entity Analytics > Privilege User Monitoring
3. Select the `File` option to add data
4. Add one of the CSV files to the open modal and upload
5. Repeat but now upload one of files with the omitted users
Alternatively, testing only the backend only is possible by directly
hitting the API wit curl
```
curl -u elastic:changeme \
-X POST "http://localhost:5601/api/entity_analytics/monitoring/users/_csv" \
-H "kbn-xsrf: true" \
-F "file=@test.csv;type=text/csv"
```
#### Verifying
Easiest way is to use the dev tools to `_search` the privmon users index
with:
```
GET .entity_analytics.monitoring.users-default/_search
```
Look for number of hits and/or use `query` to search for omitted users.
## Remaining work
- [x] API integration tests
- [ ] Batching logic unit tests
- [ ] E2E tests?
---------
Co-authored-by: machadoum <pablo.nevesmachado@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
- anonymization rules shows up correctly in advanced settings, once
registered.
- adds default rules (disabled)
- updates schema, type, and advanced settings description. removes `id`,
`description` and makes `entityClass` required for regex only
Still unregistered for now, but can [register to
view](https://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/private/observability_ai_assistant_management/server/plugin.ts#L24-L25)
To make sure the rules function without having to register the setting
(and have it show in the UI), you can run the below in dev tools:
```
POST kbn:/api/kibana/settings
{
"changes": {
"observability:aiAssistantAnonymizationRules": "[\n {\n \"type\": \"regex\",\n \"pattern\": \"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\\\.[A-Za-z]{2,}\",\n \"enabled\": true,\n \"entityClass\": \"EMAIL\"\n },\n {\n \"type\": \"regex\",\n \"pattern\": \"https?://[^\\\\s]+\",\n \"enabled\": true,\n \"entityClass\": \"URL\"\n },\n {\n \"type\": \"regex\",\n \"pattern\": \"\\\\b(?:\\\\d{1,3}\\\\.){3}\\\\d{1,3}\\\\b\",\n \"enabled\": true,\n \"entityClass\": \"IP\"\n },\n {\n \"type\": \"ner\",\n \"enabled\": true\n }\n]"
}
}
```
or with kibana config:
```
uiSettings:
overrides:
"observability:aiAssistantAnonymizationRules": |
[
{
"type": "regex",
"pattern": "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[A-Za-z]{2,}",
"enabled": true,
"entityClass": "EMAIL"
},
{
"type": "regex",
"pattern": "https?://[^\\s]+",
"enabled": true,
"entityClass": "URL"
},
{
"type": "regex",
"pattern": "\\b(?:\\d{1,3}\\.){3}\\d{1,3}\\b",
"enabled": true,
"entityClass": "IP"
},
{
"type": "ner",
"enabled": true
}
]
```
---------
Co-authored-by: Arturo Lidueña <arturo.liduena@elastic.co>
This PR guards changes to the streams state that go through
`State.attemptChanges` via the newly introduced lock manager.
If two requests are happening at the same time, one of them now fails
with a 409.
## Concerns
* Lock expiry is 30s for now - is this too little? Should be good enough
for now, maybe we need to reconsider once we introduce the bulk api
* This is only guarding changes that go through the `State` class - some
things like queries and dashboards do not, so they can still be subject
to race conditions. We could sprinkle more locks over the code base, but
I would like to solve this by moving them into `State` as well, that
seems like the cleaner approach, even though a bit more effort
* Biggest question - on this PR the concurrent request fails directly
with a 409. Is this OK or should it wait and retry a couple times? I'm
in favor of starting like this and seeing if this is actually a problem.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Kevin Lacabane <kevin.lacabane@elastic.co>
## Summary
Closes https://github.com/elastic/observability-dev/issues/4539
Fixes https://github.com/elastic/kibana/issues/221035
Enables custom roles for Observability projects in serverless.
The following is a summary of the changes:
## Feature renaming
1. Renamed `Uptime and Synthetics` to `Synthetics`
2. Renamed `APM and User Experience` to `Applications`
3. Renamed `Metrics` to `Infrastructure`
## Category reassignment
1. Changed `Dashboard` category from `Analytics` to `Observability`
2. Changed `Discover` category from `Analytics` to `Observability`
3. Changed `ML` category from `Analytics` to `Observability`
## Feature hiding
1. Hides the `Stack Alerts` feature.
2. Provides backwards compatibility for alerts created via Stack Alerts.
This enables our users to import rules created within Stack Alerts and
expect to see them in the Observability rules table.
## Navigation updates
1. Adds a `Custom Roles` link under the `Access` section in the
management navigation
2. Adds a `Manage Organization Members` link under the `Access` section
in the management navigation
3. Removes the `Users and Roles` link from the navigation footer (in
favor of the `Manage Organization Members link)
## Bug fixes
1. Fixes a bug where the `Alerts` link was not shown for Synthetics only
user (in stateful and serverless)
2. Fixes a bug where the `Alerts` link was not shown for Logs only user
(in stateful and serverless)
## Alert Override Removal
In the alerting framework, each rule is assigned a `consumer` value.
This `consumer` value changes depending on where the rule is created in
Kibana. However, in serverless we introduced an override that caused the
`consumer` value to be `Observability` in nearly every case. This logic
branched from stateful causing complexity and a large mental burden for
our engineers. Ultimately, this override became the source of bugs,
uncertainty, and unintended user experiences. Because of this, we've
removed this overrides.
If we kept this override, it would have the unfortunate side effect of
making all rules created in serverless visible from all custom roles (an
APM only user would have been can see Synthetics rules, and vice
versus). To make things more unpredictable, when users import their
rules from stateful the behavior would be different (access would be
properly mapped to the specific feature).
To address these specific user experience issues, and remove the source
of complexity, branching logic, and bugs, we removed this override logic
and restored the rule access behavior to match with stateful.
We did this while introducing backwards compatibility logic, ensuring
rules created in earlier versions of an oblt stateful cluster continue
to work and are accessible by a user with the right role access.
# Testing
1. Run local ES
```
yarn es serverless --projectType=oblt -E xpack.security.authc.native_roles.enabled=true
```
2. Run local Kibana
```
yarn start --serverless=oblt --xpack.security.roleManagementEnabled=true --xpack.cloud.users_and_roles_url="https://test_users_and_roles_url"
```
3. Login to Kibana with the admin role. Navigate to the Custom Roles
page via the management navigation.
4. Create a custom role
5. Log out of Kibana
6. Log back in with your custom role. You can do so by typing the custom
role name into the mock saml auth
<img width="460" alt="Screenshot 2025-05-22 at 9 23 13 PM"
src="https://github.com/user-attachments/assets/8e7f659b-5fe9-4e74-8c57-b420467d309e"
/>
---------
Co-authored-by: Jason Rhodes <jason.rhodes@elastic.co>
Co-authored-by: Faisal Kanout <faisal.kanout@elastic.co>
Co-authored-by: “jeramysoucy” <jeramy.soucy@elastic.co>
## Summary
Closes https://github.com/elastic/kibana/issues/222555
Remove `confidence` and `is_correction` attributes from knowledge base
entry since they are no longer used.
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: Søren Louv-Jansen <sorenlouv@gmail.com>
This PR closes#221947.
https://github.com/user-attachments/assets/25beac10-5677-42ef-9544-b3ede0bf9fa1
The last part of the video is me trying to find the telemetry event
fired when clicking the button.
**Acceptance Criteria:**
- The list should appear on the Related Dashboards tab, under the Linked
Dashboards ✅
- Users should have the ability to add or "promote" a suggested
dashboard to the list of linked dashboards stored on this rule, in a
single click ✅
- We are collecting telemetry so that we know how many times users click
the button to promote a suggested dashboard to a linked dashboard ✅
---------
Co-authored-by: Dominique Belcher <dominique.clarke@elastic.co>
## Summary
This PR adds the initial FTR test setup for the serverless Observability
Logs Essentials tier.
> [!IMPORTANT]
This PR is focused on providing the base structure and make sure the
tests can run in CI. The included tests are serving as usage examples
only and will have to be extended by Logs Essentials owners to achieve
proper test coverage.
### Changes
> [!NOTE]
The selection of tests for the logs essentials tier happens through
inclusion in specific test index files which are loaded by logs
essentials specific test config files. No tagging or other modification
of existing tests needed - the keep running for the complete tier. Test
files / directories can be included in complete tier test index files or
logs essentials tier test index files or both, giving us full control
over which tests are running where.
#### `x-pack/test_serverless`
- Add api_integration and functional config files and index files,
including common tests and load some example tests each - this way,
existing complete tier tests can be selected and included fro logs
essentials test runs.
- Add api_integration and functional `logs_essentials_only` directories
and include them in the logs essentials test index file - this way, new
test can be created that do not run on the complete tier and are
specific to the logs essentials tier.
- Add one example test each (assert the SLO settings API is not
available / assert the SLO menu entry is not displayed). These serve as
placeholders and are mainly there to demonstrate usage of this area -
going forward, they should be replaced, extended and/or modified as
needed.
#### `x-pack/test/api_integration/deployment_agnostic`
- Extend serverless base config parameters with an optional `tier` so
actual configs calling `createServerlessTestConfig` can provide a
`tier`.
- Add logs essentials specific config and index file and include core
tests as an example.
- There are no `logs_essentials_only` tests in this area, because logs
essentials is a serverless only option and as such tests specific to
that option should go into the `test_serverless` directory.
## Summary
Part of https://github.com/elastic/kibana-team/issues/1503
This PR is mostly about moving tests from
x-pack/test/alerting_api_integration
Before:
```
x-pack/test/alerting_api_integration
| - basic/
| - common/
| - observability/
| - packages/
| - security_and_spaces/
| - spaces_only/
| - spaces_only_legacy/
```
After:
```
x-pack/platform/test/alerting_api_integration
| - basic/
| - common/
| - packages/
| - security_and_spaces/
| - spaces_only/
| - spaces_only_legacy/
x-pack/solutions/observability/test/alerting_api_integration
| - observability/
x-pack/solutions/security/test/alerting_api_integration
| - security_and_spaces/group1/
| - security_and_spaces/group2/
```
Details:
- few tests in original `backfill` dir use helper function relying on
import from `@kbn/security-solution-plugin`, both tests and helpers were
relocated to `x-pack/solutions/security/test` dir.
- `observability` dir and its tests were relocated directly to
`x-pack/solutions/observability/test` dir.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Customers who enabled risk scoring in a version below 8.18 and then
upgraded to 8.18 or 9.0, will find risk scoring stops working in 0-30
days, this is because the ingest pipeline to set event ingested is not
installed, but is specified on the risk scoring index, causing all
writes to fail.
This PR fixes this by installing the ingest pipeline in all spaces where
either:
- risk scoring is set up (the risk-engine-configuration saved object is
present)
- the asset criticality index exists
This PR also increases the internal versioning of the asset criticality
and risk scoring indices, to trigger the migration which will:
- add the pipeline to the asset criticality index
- add the pipeline to a risk scoring index which hasnt been rolled over
- remove the pipeline from the risk scoring latest index if it is set
(if risk scoring was set up in 9.0 or 8.18 for the first time) we remove
this because the latest index should not modify the risk scoring
documents.
The result is that asset criticality and risk scoring should work and
event.ingested should correctly be set.
## Other important technical changes
- whenevr we increase the risk scoring internal index version, this will
now trigger a rollover on the data stream
- asset criticality and risk scoring index migrations now support
settings changes
## test steps
- on 8.17 initialise risk engine and generate some risk scores
- upgrade to 8.18.0 or 9.0.0
- rollover the risk score latest datastream dev tool `POST
/risk-score.risk-score-default/_rollover`
- verify risk scoring has broken, easy way is by assigning asset
criticality in the entity flyout
- upgrade to 9.0.3 or 8.18.3
- verify risk scoring and asset criticality work
## Test steps for local env:
- start kibana
- checkout SDG branch
https://github.com/elastic/security-documents-generator/pull/153
- run `yarn start risk-score-ingest-bug` AS SUPERUSER
- observe asset criticality assignment results in an error / is not
performed
- restart kibana
- observe the ingest pipeline has been created and it works again
dev tools verification:
```
# the pipeline should have been created
GET /_ingest/pipeline/entity_analytics_create_eventIngest_from_timestamp-pipeline-default
# the mapping version should be 4 and the default pipeline should be set
GET /.asset-criticality.asset-criticality-default
# the mapping version should be 4 and the default pipeline should be set
GET /risk-score.risk-score-default
# the default pipeline should be set
GET /_index_template/.risk-score.risk-score-default-index-template
# the default pipeline should NOT be set
GET /risk-score.risk-score-latest-default
```
Other verifications:
- risk scoring should work
- asset criticality assignment should work
- ???
---------
Co-authored-by: machadoum <pablo.nevesmachado@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Re-submit of https://github.com/elastic/kibana/pull/216352 as it has
merge conflicts and we don't have write permissions for Sandra's remote.
To test, add the following to your kibana.yml:
```
uiSettings:
overrides:
"observability:aiAssistantAnonymizationRules":
- id: "ner"
type: "ner"
enabled: true
- id: "beach"
type: "regex"
enabled: true
pattern: "sandy"
```
---------
Co-authored-by: Sandra Gonzales <sandra.gonzales@elastic.co>
Co-authored-by: Sandra G <neptunian@users.noreply.github.com>
We've been seeing intermittent timeouts over the last few weeks waiting
for package manifests to load.
Successful runs are also near 2 minutes, which seems to indicate this
isn't an issue with the service starting up.
Resolves https://github.com/elastic/kibana/issues/218168
## Summary
Sorts Dashboards by how relevant they are to the current alert using
jaccard similarity.
Limits suggestions to 10 results.
### Testing
I'm not expecting this to be manually tested, as the API is not
currently in use by the UI. Automated testing has been included to test
this feature.
Adds a new processor to streams that serves as an escape hatch to
regular Elasticsearch ingest pipelines (entered as JSON).
<img width="1139" alt="Screenshot 2025-05-28 at 15 54 41"
src="https://github.com/user-attachments/assets/67f1f4c4-982e-45d1-ae96-080545c5a0e2"
/>
Some details:
* If `on_failure` or `ignore_failure` are set via the JSON input, the
"outer" definition or ignore_failure flag are ignored
* Expands to multiple processors in the ingest pipeline
* Does minimal validation (needs to be one of a list of known existing
Elasticsearch processors based on the Elasticsearch API types), but
doesn't enforce valid Elasticsearch processors otherwise
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Relocating tests added in #221139 into separate feature flag configs,
that are not run on MKI but only Kibana CI.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Enable ILM for classic streams.
We can now overwrite template settings at the data stream level with
`PUT _data_stream/{name}/_settings` so we can enable full lifecycle
capabilities on classic stream.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Part of https://github.com/elastic/kibana-team/issues/1503
This PR is mostly about moving tests from x-pack/test to platform and
solutions dirs
Before:
```
x-pack/test/api_integration
| - apis/metrics_ui/
| - apis/ml/
| - apis/entity_manager/
| - apis/search_playground/
```
After:
```
x-pack/platform/test/
| - apis/entity_manager/
| - apis/ml/
x-pack/solutions/observability/test/
| - apis/metrics_ui/
x-pack/solutions/search/test/
| - apis/search_playground/
```
New test package was created under `x-pack/solutions/search/test` for
Search solution tests.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This PR moves the migration-on-read logic into the storage adapter so
it's not possible to accidentally read unmigrated `_source` objects and
adds tests for this for asset links (dashboards and queries). While
there is no migration on read currently ongoing for dashboards and
queries, this test will highlight problems in this area, avoiding an
issue like the one we had with streams definitions themselves.
## Summary
- API test that verifies whether fields are correctly inherited
- update an assertion that fails on Cloud because a cluster may have
existing streams in place
The handling of `.kibana_streams` documents in the execution plan wasn't
optimal because it would update the definition docs of ancestors and
descendants of all changed streams as well, even if no change actually
happened.
This PR fixes this problem by checking whether a stream got actually
changed or not.
It also switches the bulk operation for changed definitions from the
implicit `"wait_for"` refresh param to `true`. Since concurrent access
is anyways not supported by the streams API (there are plans to
introduce an explicit lock), this won't change make things less
efficient in practice but return the request faster to the user.
### Summary
Sets the priority of the `@stream` templates to the highest possible
value which guarantees that no existing templates can take precedence if
it is successfully created. Since no overlapping templates can have the
same priority, the template will fail to create if there is an existing
template that covers the stream index pattern.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
issue: https://github.com/elastic/security-team/issues/12309
Introduces the integrations card in the SIEM migrations topic, after the
"start rule migration" card.
Requirements:
- The same integrations card UI as the rest of the onboarding topics.
- When no migration is complete, show the same recommended integrations
as the default topic.
- When some migration is complete, show the integrations detected by the
rule migrations (all of them).
- Render a label next to the title showing the number of migration rules
linked to the integration (both prebuilt and custom rules)
- Sort the integrations list by the number of rules linked.
- As the last integration item in the list, render a card to link the
integration auto-import
### Screenshots
When no migration has been executed yet:
When some migrations have been completed:
Auto import card is appended at the end:
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This PR extends streams to allow mixed otel/ecs querying.
Walkthrough (slightly outdated):
https://github.com/user-attachments/assets/9d6280ff-57b2-43d8-9c81-dfbfabb64585
<img width="492" alt="Screenshot 2025-03-07 at 17 15 27"
src="https://github.com/user-attachments/assets/792fbbc2-6cbe-42fe-96b3-eea2b6959fe5"
/>
# To test
* Start up PR, enable streams via `POST kbn:/api/streams/_enable`
* Redirect all incoming logs to streams:
```
PUT _ingest/pipeline/logs@custom
{
"processors": [
{
"reroute": {
"destination": "logs"
}
}
]
}
```
* Send some logs
* They will be trandslated to namespaced ECS and can be queried the same
way
Some demo actions:
```
# Make sure trial license is active (because of synthetic source)
# Enable streams
POST kbn:/api/streams/_enable
# Fork some data to be able to edit mapping and stuff
POST kbn:/api/streams/logs/_fork
{
"stream": { "name": "logs.child" },
"if": {
"field": "attributes.custom_field",
"operator": "exists"
}
}
# Routing could have been done on "custom_field" as well
# Send some ECS-formatted data
POST logs/_doc
{
"message": "Hello world",
"log.level": "warn",
"host": {
"name": "my-mbp",
"ip": "1.2.3.4"
},
"custom_field": "value2"
}
# Send some OTel-formatted data
POST logs/_doc
{
"body": {
"text": "Hello OTEL world"
},
"severity_text": "warn",
"resource": {
"attributes": {
"host.name": "my-mbp",
"host.ip": "1.2.3.4"
}
},
"attributes": {
"custom_field": "value"
}
}
# See how it's stored as namespaced ECS (beware automatic aliasing)
GET logs.child/_search
# See the fields and their aliases
GET kbn:/api/streams/logs.child
# Nested data treatment (objects flattened, arrays kept)
POST logs/_doc
{
"message": "Hello world",
"log.level": "warn",
"host": {
"name": "my-mbp",
"ip": "1.2.3.4"
},
"custom_field": "value2",
"nested_custom_field": {
"deeply": {
"deeply": {
"nested": "value"
}
}
},
"array_values": [
{ "deeply": { "nested": 123 }},
{ "deeply": { "nested": 456 }}
]
}
GET logs.child/_search
# Map a field (show alias in app)
PUT kbn:/api/streams/logs.child/_ingest
{
"ingest": {
"lifecycle": {
"inherit": {}
},
"processing": [],
"routing": [],
"wired": {
"fields": {
"attributes.custom_value": {
"type": "keyword"
}
}
}
}
}
# Trying to map "custom_value" will result in an error - only attributes.* and resource_attributes.* can be mapped
# End experiment
POST kbn:/api/streams/_disable
```
## Do on separate PRs
* More dedicated integration tests
* Replace the painless implementation of normalization with an
Elasticsearch processor (not shipped yet)
* Remove the dot expander processors once Elasticsearch allows accessing
flattened field names
---------
Co-authored-by: Chris Cowan <chris@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Background
The semantic text migration was added in 8.17 along with the move to
semantic_text field (https://github.com/elastic/kibana/pull/186499).
## Suggestion
We should remove the semantic_text migration starting in 9.1. Reasons:
- The migration has recently caused severe disruption on a number of
clusters (https://github.com/elastic/kibana/pull/220255).
- The migration may no longer be needed. Users are required to upgrade
Elasticsearch to 8.18 in order to update to 9.0, or 8.19 to upgrade to
9.1. This could mean that users are guaranteed to have run the migration
previously, when upgrading Kibana to 9.1 (needs to be double checked
with Kibana folks)
---------
Co-authored-by: Viduni Wickramarachchi <viduni.ushanka@gmail.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
When introducing package registry configuration in
https://github.com/elastic/kibana/pull/193144 I missed the set up of
`xpack.fleet.registryUrl` which has resulted in flaky tests whenever the
test uses fleet underneath.
## Summary
Verifies that the uncompressed size of a single object included in the
content pack do not exceed 1MB as a safety measure. The 1MB limit is
based on MAX(`elastic/integrations` dashboard size) * 2.
The change also includes a constraint on the archive internal structure
and expects a single root directory as entry.
Example:
```
content_pack-1.0.0/manifest.yml
content_pack-1.0.0/kibana/dashboard/123-..json
content_pack-1.0.0/kibana/index-pattern/123-..json
...
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
