Closes: https://github.com/elastic/kibana/issues/220954
`fleet/remote_synced_integrations/<output-id>/remote_status` includes
non-http request errors in the error field of the status response for
improved client handling.
## Summary
Resolves#222051.
This PR makes a richer empty state UX for the Investigation Guide
feature we added to the Alert Details page.
Before, when a rule did not have an investigation guide, the tab on the
alert details page dedicated to the feature was disabled. Now, the tab
is always enabled. When an investigation guide is present on a rule, a
badge will display to indicate the tab contains content.
If the user clicks into the tab when the rule does not have an
investigation guide, they will see an empty state with a call to action
to create a guide. If the user decides to click the empty state button,
it will open the Rule Edit flyout. I have added additional functionality
that allows the flyout to take an `initialStep` prop, so we pre-set the
flyout to the `Details` step which contains the text field the user can
use to create their guide.
The copy, iconography, and layout of the tab heading are all in draft
state pending some design feedback. I will also add some tests to the
code that I have added.
## Reviewing this PR
_Note to technical writers:_ You can see the copy added for the empty
state
[here](https://github.com/elastic/kibana/pull/223974/files#diff-71b439414e4974e2decb0f25c136f52ccea4b49ebe393af68dfc5fd184d56e1cR37).
Here's a screenshot as well:
<img width="375" alt="image"
src="https://github.com/user-attachments/assets/491d87ac-b473-484e-82cd-45a1bd197c61"
/>
### Technical review
1. Create a rule that will generate alerts; _do not_ define an
Investigation Guide for it
1. Trigger an alert, and go to the details page
1. You should see the Investigation guide tab is available, whereas on
`main` it would be disabled
1. Open the Investigation Guide tab, you should see the empty state with
its CTA
1. Click this button, the rule flyout should open in Edit mode, and the
Details step should be pre-selected
1. Define an investigation guide, this can be any text. Save the rule.
1. Once you have saved the rule you should see the rule data update in
the page. The empty state will be gone and the investigation guide you
defined will be there instead.
## Summary 🌷
Resolves https://github.com/elastic/observability-dev/issues/4568
### Test Now flyout
Checked this by throwing an error within the flyout:
<img width="1584" alt="image"
src="https://github.com/user-attachments/assets/d8cfe96e-8b1b-4c28-819e-82411b5f20af"
/>
### Monitor detail flyout
In the case of this flyout, IMO, we should not use an error boundary. We
are receiving an error that has been caught and placed into the plugin's
Redux store, and then selecting that as part of the render procedure of
this component. Thus, we should display it using `EuiCallout`, which is
the recommended course per [the
docs](https://eui.elastic.co/docs/patterns/error-messages/error-banners/).
Example implementation below:
_note:_ all the copy here is placeholder and we should run it by our
tech writers.
<img width="607" alt="image"
src="https://github.com/user-attachments/assets/ef60258e-e50e-4fa7-adc5-b77469ddc0fb"
/>
We should not use an error boundary, as they are there specifically to
handle unforeseen errors that we can't account for at development time.
In this case, we have already caught the error and persisted it in app
state, so I don't think it's appropriate to display it in an error
boundary as the runtime is still progressing and is not broken.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Fixes tab selection issue when user opens alert details page from
"Related alerts" tab. In this case, "Overview" tab should be selected by
default. In other cases, tab Id passed from URL should be used otherwise
default to "Overview".
## Summary
- Adds configuration for the product intercept in `oblt`, `es` and
`security` serverless offerings, alongsides stateful offering too. The
configuration provided sets the intercept to display every 90days, this
is configurable through the config `xpack.product_intercept.interval`.
The intercept can also be turned off through the config
`xpack.product_intercept.enabled`
- Also tweaks prompter timer implementation to accommodate inherent
[issue with long timer
delays](https://developer.mozilla.org/en-US/docs/Web/API/Window/setTimeout#maximum_delay_value)
in the browser
- Adjusts the signature of the `registerIntercept` method, such that a
deferred value to be evaluated when the intercept is to be displayed is
passed. This unlocks the ability to have consumers provide dynamically
imported modules that provide the config for the intercept, see
0e07892217
for an example.
### How to test
- Add the following config to your `kibana.dev.yml` file;
```yml
xpack.product_intercept.enabled: true
# we set the interval to 30s so the wait long period to display the
intercept is bearable
xpack.product_intercept.interval: '30s'
```
- Start kibana in stateful, and serverless mode, in either scenario you
should be presented the product intercept, with the intercept
specifically stating the current product the user is interacting with.
See below for an example of observability solution;
https://github.com/user-attachments/assets/6ca6baf2-58d3-4002-ac94-ec6e9a0902ae
<!--
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.
Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.
- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...
-->
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Resolves https://github.com/elastic/kibana/issues/222104
## Summary
Adds optional flag when registering a rule type for "dangerously
creating alerts in all spaces". If a rule type opts into this flag,
alerts created during rule execution will persist the `kibana.space_ids`
field as `"*"` instead of the space ID of the rule. Note that we store
`kibana.space_ids` as a string array, so the final alert document will
have
```
'kibana.space_ids': ['*']
```
This PR just adds the flag and updates the code to respect the flag. It
does not opt any rule types into using it. You can look at the
functional tests to see example test rule types that use it.
Because the streams rule type that we expect to be the first user of
this flag uses the `persistenceRuleTypeWrapper` in the rule registry for
writing alerts, we also had to update the rule registry code.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Addresses a bug with the `ConnectorTokenClient` when using the
`UnsecuredActionClient` to execute actions directly (vs enqueuing a task
for execution). We previous pass in an internal saved objects repository
(that doesn't require a user request) to the `ConnectorTokenClient` when
using the `UnsecuredActionsClient` but this does not create the
`connector_token` saved object correctly so the next time it's read, we
get a `Failed to decrypt attribute` error.
This only occurs when using the `sendAttachmentEmail` with the MS
Exchange connector function added in this PR:
https://github.com/elastic/kibana/pull/219164. It does not affect the
other email service methods.
## To Verify
1. Ask me for MS Exchange credentials
2. Add this to your Kibana config:
```
xpack.actions.preconfigured:
test-exchange-email:
name: preconfigured-exchange-email
actionTypeId: .email
config:
service: exchange_server
clientId: <clientId>
tenantId: <tenantId>
from: <from>
secrets:
clientSecret: <secret>
notifications.connectors.default.email: test-exchange-email
```
3. Make this change to the code so Kibana sends 2 emails when it starts
up:
```
--- a/x-pack/platform/plugins/shared/notifications/server/plugin.ts
+++ b/x-pack/platform/plugins/shared/notifications/server/plugin.ts
@@ -40,6 +40,27 @@ export class NotificationsPlugin
public start(_core: CoreStart, plugins: NotificationsServerStartDependencies) {
const emailStartContract = this.emailServiceProvider.start(plugins);
+ const emailService = emailStartContract.getEmailService();
+ emailService
+ .sendAttachmentEmail({
+ to: ['<email>'],
+ subject: 'yo',
+ message: 'i am here',
+ attachments: [],
+ spaceId: 'default',
+ })
+ .then(() => {
+ new Promise((resolve) => setTimeout(resolve, 5000)).then(() => {
+ emailService.sendAttachmentEmail({
+ to: ['<email>'],
+ subject: 'yo',
+ message: 'i am here again',
+ attachments: [],
+ spaceId: 'default',
+ });
+ });
+ });
+
return {
```
4. Verify there are no decryption errors for the `connector_token` SO
logged and that the emails are sent successfully.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
This is a farewell PR to Cases. Probably my last PR to the cases
codebase. It was quite a journey, and I learned a lot. I hope the best
for the feature of Cases.
## Decisions
Just before Cases was forbidden to do migrations, we did a last
migration to all cases to persist `total_alerts: -1` and
`total_comments: -1`. We did that so that in the future, when we would
want to populate the fields, we would know which cases have their fields
populated and which do not. In this PR, due to time constraints and
criticality of the feature, I took the following decisions:
- Cases return from their APIs the total comments and alerts of each
case. They do that by doing an aggregation, getting the counts, and
merging them with the response. I did not change that behavior. In
following PRs, it can be optimized and fetch the stats only for cases
that do not yet have their stats populated (cases with -1 in the counts)
- When a case is created, the counts are zero.
- When a comment or alert is added, I do an aggregation to get the stats
(total alerts and comments) of the current case, and then update the
counters with the number of the newly created attachments. The case is
updated without version checks. In race conditions, where an attachment
is being added before updating the case, the numbers could be off. This
is a deliberate choice. It can be fixed later with retries and version
concurrency control.
- The case service will continue to not return the `total_alerts` and
`total_comments`.
- The case service will accept the `total_alerts` and `total_comments`
attributes to be able to set them.
Fixes: https://github.com/elastic/kibana/issues/217636
cc @michaelolo24
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
This PR contains the following updates:
| Package | Type | Update | Change |
|---|---|---|---|
| [papaparse](https://www.papaparse.com/)
([source](https://redirect.github.com/mholt/PapaParse)) | dependencies |
patch | [`^5.5.2` ->
`^5.5.3`](https://renovatebot.com/diffs/npm/papaparse/5.5.2/5.5.3) |
---
### Release Notes
<details>
<summary>mholt/PapaParse (papaparse)</summary>
###
[`v5.5.3`](e3c7b2628c...a4f8b0f1e3)
[Compare
Source](e3c7b2628c...a4f8b0f1e3)
</details>
---
### Configuration
📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR has been generated by [Renovate
Bot](https://redirect.github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40MjUuMSIsInVwZGF0ZWRJblZlciI6IjM5LjEwNy4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJUZWFtOiBTZWN1cml0eVNvbHV0aW9uIiwiYmFja3BvcnQ6YWxsLW9wZW4iLCJyZWxlYXNlX25vdGU6c2tpcCJdfQ==-->
Co-authored-by: elastic-renovate-prod[bot] <174716857+elastic-renovate-prod[bot]@users.noreply.github.com>
Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com>
## Rule gaps callout
### Summary
This PR introduces a new callout on the **Rules** page and improves the
**Gaps** panel with clearer metrics and filtering logic to enhance the
user experience when managing gap filling.
---
### Key Changes
1. **Rules Page – Callout for Recent Gaps**
- A dismissible callout is displayed if there are any unfilled gaps in
the **last 24 hours**.
- The callout includes:
- A link to the Gaps dashboard.
- A link to relevant documentation.
- Once dismissed, the callout won’t reappear for 24 hours **unless** new
gaps are detected during that time.
2. **Gaps Panel – Rule Count Display**
- Displays two separate counts:
- The **number of rules with unfilled gaps**.
- The **number of rules currently in progress** (gaps are being filled).
- Provides users with a quick overview of the current gap state.
3. **Filtering Logic**
- The filter now only shows **rules with unfilled gaps that can still be
filled**.
- Example: If 5 rules had unfilled gaps and filling started for 3, the
filtered list will only show the remaining 2.
**Note**: we don't immediately update number for rules with gap
"in-progress", because gap update async and can happen after bulk
request if finished.
https://github.com/user-attachments/assets/3c913129-16c3-449a-9847-a01e2657e9aa
### How to test
Enable FF `storeGapsInEventLogEnabled`.
[Use this tool to create a lot of rules with/without
gaps](https://github.com/elastic/security-documents-generator)
```
// will create 100 without gaps, and 5 minute interval
yarn start rules --rules 100 -g 0 -c -i "5m" -f 1
```
Alternatively you can create a rule with small interval (1m) and
lookback time(1s). Enable rule, wait for completion and then disable it.
Wait 5m (> 4x interval). Then enable it and rule should fail and
generate gap.
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
- Moved `search_navigation` to `plugins` root
- `search_navigation` being inside the search_solution folder was a
pre-cursor to the solutions/search migration. The introduction of the
solutions/search folder made the search_solution folder redundant,
therefore this migrates the search_navigation plugin up with all the
other search plugins.
- Moved the base classic nav items list to the search_navigation plugin.
The classic nav items are now a static list and can be servered from the
search_navigation plugin instead of being shared from
`enterprise_search`. This is mainly a devx improvement to allow all
search related navigation changes to eventually be maintained in the
search navigation plugin.
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Changes are in support of space awareness:
- Updated response action service utility for retrieving action requests
(list) so that response actions with a tag of
`INTEGRATION-POLICY-DELETED` will be returned when in a space that
matches the space defined in the `ORPHAN-RESPONSE-ACTIONS-SPACE` ref.
data key
- Updated response action service utility that retrieve a single action
to allow returning an action request with tag
`INTEGRATION-POLICY-DELETED` if the active space matches the space
defined in `ORPHAN-RESPONSE-ACTIONS-SPACE` ref. data key
- `ensureActionRequestsIndexIsConfigured()` was adjusted to be more
robust at checking index for missing mappings
## Summary
This PR changes the "soft delete" mechanism in the CSV upload to use
scripted updates.
There were 2 main reasons for this:
1. We need to check if a privileged user has been added from some other
data source
In this case, not including the user in the CSV upload shouldn't delete
it, instead only remove the `"csv"` source label.
2. The soft delete needs to take into account the full list of users
being uploaded, and not only the current batch
This means we need to collapse the stream first and then run the soft
delete logic. Doing this allows the soft delete to search the Privmon
index for all users not included in the full csv file
## How to test
#### Prerequisite
Make sure you have a CSV file with usernames
Check
[here](https://gist.github.com/tiansivive/0be2f09e1bb380fdde6609a131e929ed)
for a little helper script
Create a few copies where some of the users are deleted, in order to
test soft delete.
Make sure to create files that are over the batch size (`100`) or change
the batch size in the code.
1. Start up kibana and ES
2. Navigate to Security > Entity Analytics > Privilege User Monitoring
3. Select the `File` option to add data
4. Add one of the CSV files to the open modal and upload
5. Repeat but now upload one of files with the omitted users
Alternatively, testing only the backend only is possible by directly
hitting the API wit curl
```sh
curl -u elastic:changeme \
-X POST "http://localhost:5601/api/entity_analytics/monitoring/users/_csv" \
-H "kbn-xsrf: true" \
-F "file=@test.csv;type=text/csv"
```
#### Verifying
Easiest way is to use the dev tools to `_search` the privmon users index
with:
```
GET .entity_analytics.monitoring.users-default/_search
```
Look for number of hits and/or use `query` to search for omitted users
like:
```json
{
"query": {
"bool": {
"must": [
{ "term": { "labels.monitoring.privileged_users": "deleted" } }
]
}
}
}
```
Verify that the `"deleted"` users are _only_ the ones missing from the
last uploaded file.
## Summary
This PR introduces Fleet’s agent space awareness to the Osquery plugin.
The primary goal is to ensure that agents, policies, packs, and queries
remain isolated to their respective Kibana spaces. Specifically:
1. Agents enrolled in policy A (belonging to Space A) are not accessible
from Space B—live queries cannot be performed on them from other spaces.
2. Packs created in Space A are not visible in Space B.
3. Saved queries created in Space A are not visible in Space B.
4. Agent policies, agents, and space-exclusive operating systems from
Space A are not visible from Space B.
5. Live query history is only visible in a space if it includes agents
from that space.
## Implementation Details
- All usages of the Saved Object Client have been updated to use the
space-aware SO client, ensuring queries only return results from the
current space.
- When creating live queries, we now add a `space_id` field to
`.logs-osquery_manager.actions-default` documents. This field stores the
integration policy (osquery_manager) IDs of the agents involved in each
query.
- When reading live queries, searches are filtered to include only
documents with `policy_ids` that are present in the current space.
This ensures strict separation of Osquery data and actions across
different spaces.
## Feature Flag Considerations
At this stage, we believe no feature flag or gating is necessary. In all
cases, we default to the `default` space ID. We have tested these
changes with the feature flag turned off and observed the following:
| Scenario | Result |
|------------------------------------------------------|:-----------:|
| Agents and policies are visible in the Live query UI |
✅ |
| Users can query different agents/policies | ✅ |
| Live queries are saved in the live query history | ✅
|
| Users can create both packs and saved queries | ✅ |
| Users can run packs and saved queries | ✅ |
| Users can enable packs for interval execution | ✅ |
Based on these results, we conclude that this PR can be safely merged
without enabling the feature flag, as it does not impact current
operations.
**The index mappings for `.logs-osquery_manager.actions-default` will be
updated to introduce the `policy_ids` field. This field will also be
populated when actions are created. This change is fully backward
compatible and does not introduce any breaking changes.**
### Enable feature flags
```yaml
xpack.securitySolution.enableExperimental:
- endpointManagementSpaceAwarenessEnabled
xpack.fleet.enableExperimental:
- useSpaceAwareness
```
### Switch Fleet to Space aware
```http request
POST /internal/fleet/enable_space_awareness
Elastic-Api-Version: 1,
```
## Manual Test Cases for Space Awareness (Feature Enabled)
### Setup
- **Space A:** Policies `PolicyA1`, `PolicyA2`, `PolicyAB` (each with an
enrolled agent)
- **Space B:** Policies `PolicyB1`, `PolicyB2`, `PolicyAB` (each with an
enrolled agent)
- `PolicyAB` is present in both spaces
---
### 1. Agent Visibility
| Test Step | Expected Result | Pass |
|---------------------------------------------------------------------------|------------------------------------------------------|----------|
| In Space A, view the list of agents and policies | Only agents for
`PolicyA1`, `PolicyA2`, `PolicyAB` are visible | ✅ |
| In Space B, view the list of agents and policies | Only agents for
`PolicyB1`, `PolicyB2`, `PolicyAB` are visible | ✅ |
| `PolicyAB`'s agents are visible in both spaces | Agents enrolled in
`PolicyAB` are visible in both spaces | ✅ |
---
### 2. Live Query Permissions
| Test Step | Expected Result | Pass |
|---------------------------------------------------------------------------|------------------------------------------------------|----------|
| In Space A, attempt a live query on an agent from `PolicyB1` or
`PolicyB2`| Agent is not available for selection/query |
✅ |
| In Space B, attempt a live query on an agent from `PolicyA1` or
`PolicyA2`| Agent is not available for selection/query |
✅ |
| In either space, perform a live query on an agent from `PolicyAB` |
Query is allowed and succeeds | ✅ |
| In Space A, attempt a live query on an agent from `PolicyA1` or
`PolicyA2`| Query is allowed and succeeds | ✅ |
| In Space B, attempt a live query on an agent from `PolicyB1` or
`PolicyB2`| Query is allowed and succeeds | ✅ |
---
### 3. Packs and Saved Queries
| Test Step | Expected Result | Pass |
|---------------------------------------------------------------------------|------------------------------------------------------|----------|
| In Space A, create a pack and a saved query | Only visible in Space A
| ✅ |
| In Space B, create a pack and a saved query | Only visible in Space B
| ✅ |
| In Space A, check for packs/queries created in Space B | Not visible |
✅ |
| In Space B, check for packs/queries created in Space A | Not visible |
✅ |
| In Space A, live query a pack that has Agent A and Agent AB. | In
space B user doesnt see results from Agent A | ✅ |
---
### 4. Live Query History
| Test Step | Expected Result | Pass |
|---------------------------------------------------------------------------|------------------------------------------------------|----------|
| In Space A, view live query history | Only queries involving agents
from Space A (including `PolicyAB`) are visible | ✅ |
| In Space B, view live query history | Only queries involving agents
from Space B (including `PolicyAB`) are visible | ✅ |
---
### 5. Pack Execution
| Test Step | Expected Result | Pass |
|---------------------------------------------------------------------------|------------------------------------------------------|----------|
| In Space A, run a pack on agents from `PolicyA1`, `PolicyA2`,
`PolicyAB` | Execution succeeds | ✅ |
| In Space B, run a pack on agents from `PolicyB1`, `PolicyB2`,
`PolicyAB` | Execution succeeds | ✅ |
| Attempt to run a pack on an agent from another space | Not possible;
agent not available for selection | ✅ |
---
### 6. Interval Execution of Packs
| Test Step | Expected Result | Pass |
|---------------------------------------------------------------------------|------------------------------------------------------|----------|
| In Space A, enable interval execution for a pack | Only
agents/policies in Space A are affected | ✅ |
| In Space B, enable interval execution for a pack | Only
agents/policies in Space B are affected | ✅ |
closes https://github.com/elastic/kibana/pull/222935
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
> [!TIP]
> looks huge, but
> - 5,402 lines snapshot tests
> - 714 lines yaml files
## Summary
This PR adds a new feature version `siemV3` with the required role
migrations, in order to enable the new privilege
`global_artifact_management_all` for users where needed.
### What's in the PR?
- Required changes around security role migration from `siemV2` to
`siemV3`
- Improvements by parameterizing `siemV3` in lots of places, to ease
future role migrations by decreasing the occurrences that have to be
changed.
- A new function called `baseFeatureConfigModifier()` in
`ProductFeaturesConfig`: now product features have the ability to modify
the base Kibana feature. de05a3b167
- Product feature `endpointArtifactManagement` is split to
ESS/Serverless counterparts, and adds role migrations to the base Kibana
config using `baseFeatureConfigModifier()`
(1c31f56b43). This solves 2 problems:
- Different migrations are needed for ESS and Serverless.
- The product feature `endpointArtifactManagement`, and hence the
privilege `global_artifact_management_all` is not available on all
serverless tiers (see [these
fails](https://buildkite.com/elastic/kibana-pull-request/builds/310534/summary/annotations?jid=019788c8-d03e-44e7-867f-ff1557f9e894#019788c8-d03e-44e7-867f-ff1557f9e894/256-4872)),
therefore the migration needed to be separated from the base Kibana
feature config.
- (note: these changes were safeguarded by the role migration tests and
snapshot tests)
- Security / **Global Artifact Management** [space awareness]:
- moves the sub-privilege out of feature flag, in order to be able to
target it for role migrations
- adds a 'Coming soon' test to the privilege
- `endpointManagementSpaceAwarenessEnabled:false`
<img width="500" alt="image"
src="https://github.com/user-attachments/assets/fdfd5fc7-7f7d-4210-96c9-09e2357530c0"
/>
- `endpointManagementSpaceAwarenessEnabled:true`
<img width="500" alt="image"
src="https://github.com/user-attachments/assets/f8361a4c-da6e-416c-b728-5788eb1a053e"
/>
- role migration is added: in short, any artifact ALL privilege causes
the new Global Artifact Management ALL privilege to be added
(https://github.com/elastic/security-team/issues/11717)
- predefined roles are updated locally
(note: in elasticsearch-controller, it'll be updated after this PR is
merged and deployed,
https://github.com/elastic/elasticsearch-controller/pull/1010)
- tests!
- testing the migration itself: b8d90d085f
and 309abb30a8
- snapshot test with deprecated features:
https://github.com/elastic/kibana/pull/219566/files#diff-ed11536475a7a6f0a835cbc950c3b7405093058ad42bab30cf06f41ed21561a3
- some functional tests enabled for deprecated features:
4b4f49ea3d
## Global Artifact Management role migrations
```mermaid
flowchart LR
subgraph siemV2[siem/siemV2]
none1[none]
end
subgraph siemV3
none2[none]
end
none1 --> none2
```
```mermaid
flowchart LR
subgraph siemV2[siem/siemV2]
read1[read]
end
subgraph siemV3
read2[read]
end
read1 --> read2
```
```mermaid
flowchart LR
classDef serverless stroke:blue,stroke-dasharray: 5 5
subgraph siemV2[siem/siemV2]
subgraph minread1[minimal_read]
minread1_subs["`trusted_applications_read
event_filters_read
blocklist_read
host_isolation_exceptions_read`"]
minall1_subs["`trusted_applications_all
event_filters_all
blocklist_all
host_isolation_exceptions_all`"]
eer1["`endpoint_exceptions_read
(only serverless)`"]:::serverless
eea1["`endpoint_exceptions_all
(only serverless)`"]:::serverless
end
end
subgraph siemV3
subgraph minread2[minimal_read]
minread2_subs["`trusted_applications_read
event_filters_read
blocklist_read
host_isolation_exceptions_read`"]
minall2_subs["`trusted_applications_all
event_filters_all
blocklist_all
host_isolation_exceptions_all`"]
eer2["`endpoint_exceptions_read
(only serverless)`"]:::serverless
eea2["`endpoint_exceptions_all
(only serverless)`"]:::serverless
g2[global_artifact_management_all]
end
end
minread1 --> minread2
minread1_subs -->|each to his own| minread2_subs
minall1_subs -->|global for any of these| g2
minall1_subs -->|each to his own| minall2_subs
eer1 -->|only serverless| eer2
eea1 -->|only serverless| eea2
eea1 -->|only serverless| g2
linkStyle 4,5,6 stroke:#00f,color:blue
```
notes for above:
- `global_artifact_management_all` have to be added for **any** artifact
write privilege (trusted apps, event filters, blocklists, host isolation
exceptions)
- on serverless, there is a separate endpoint exceptions privilege, it
counts as an artifact
```mermaid
flowchart LR
subgraph siemV2[siem/siemV2]
all1[all]
end
subgraph siemV3
subgraph minall2[minimal_all]
g1[global_artifact_management_all]
end
end
all1 -->|keep access to the included Endpoint Exceptions ALL| g1
all1 -->|enable sub-feature toggle| minall2
```
notes for above:
both on serverless and ESS, Endpoint Exceptions are included in ALL,
hence the migration
> [!note]
> `siem` sub-privileges are not included in READ/ALL parent privileges.
The user needs to enable them one-by-one after enabling the sub-feature
privileges toggle. So Endpoint Exception here is an exception. In any
sense of the word.
```mermaid
flowchart LR
classDef serverless stroke:blue,stroke-dasharray: 5 5
subgraph siemV2[siem/siemV2]
subgraph minall1[minimal_all]
minread1_subs["`trusted_applications_read
event_filters_read
blocklist_read
host_isolation_exceptions_read`"]
minall1_subs["`trusted_applications_all
event_filters_all
blocklist_all
host_isolation_exceptions_all`"]
eer1["`endpoint_exceptions_read
(only serverless)`"]:::serverless
eea1["`endpoint_exceptions_all
(only serverless)`"]:::serverless
end
end
subgraph siemV3
subgraph minall2[minimal_all]
minread2_subs["`trusted_applications_read
event_filters_read
blocklist_read
host_isolation_exceptions_read`"]
minall2_subs["`trusted_applications_all
event_filters_all
blocklist_all
host_isolation_exceptions_all`"]
g2[global_artifact_management_all]
eer2["`endpoint_exceptions_read
(only serverless)`"]:::serverless
eea2["`endpoint_exceptions_all
(only serverless)`"]:::serverless
end
end
minall1 -->|only on ESS to keep access to the included Endpoint Exceptions ALL| g2
minall1 --> minall2
minread1_subs -->|each to his own| minread2_subs
minall1_subs -->|global for any of these| g2
minall1_subs -->|each to his own| minall2_subs
eer1 -->|only serverless| eer2
eea1 -->|only serverless| eea2
eea1 -->|only serverless| g2
linkStyle 5,6,7 stroke:#00f,color:#00f
linkStyle 0 stroke:#0a0,color:#0a0
```
notes for above:
when sub-feature privileges are enabled,
- on ESS endpoint exceptions are still automatically included, that's
why we need to add global access
- on serverless, endpoint exceptions are controlled by the sub-feature
privilege (just like all other artifact privileges, see the note above)
## Background
- Previous role migration PR:
https://github.com/elastic/kibana/pull/201780
- Role migration description:
https://github.com/elastic/kibana/pull/186800
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
After merging [this PR](https://github.com/elastic/kibana/pull/210769)
that updates Playwright to 1.50.1 we started to see failures in
onboarding e2e tests. After an investigation, turns out that playwright
1.50.1 has an issue where `test` CLI command is not exposed and we use
that command to run the tests.
This change updates playwright to 1.53.1 which returns the `test` CLI
command.
Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
## Summary
This PRs builds on top of the [extension point added to the E|QL
editor](https://github.com/elastic/kibana/pull/221474) by registering
recommended ES|QL queries from the Observability plugin.
It registers two recommended queries for logs and two for metrics:
- Kubernetes pods sorted by memory usage
- Kubernetes pods sorted by CPU usage
- Logs with "error" or "warn" messages
- Error occurrences by host name
### Demo - Solution navigation enabled
https://github.com/user-attachments/assets/ca06b89b-5574-4775-8beb-157fa1eaa97a
### Demo - Solution navigation disabled
Only the default recommended ES|QL queries are enabled in this mode.
https://github.com/user-attachments/assets/52043076-6467-44fa-a847-937d3e723afc
## Testing instructions
There is an environment deployed with `/oblt-deploy` that has logs and
metrics (check last month in Discover), you can find the credentials in
the issue posted by the bot in the issue timeline below
For local testing:
- Ingest logs and metrics. This can be done by running a Synthtrace
scenario, e.g. `node scripts/synthtrace.js logs_traces_hosts`
- Enable the Observability solution navigation
- Start writing an ES|QL query targeting indices that match the `logs-*`
or `metrics-*` patterns
## Open questions
- Should the `KQL` command be used in the queries?
Closes https://github.com/elastic/kibana/issues/224027
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Automatically maps ECS fields to Otel semantic conversion.
Mapping logic is pulled from
https://www.elastic.co/guide/en/ecs/master/ecs-otel-alignment-details.html
using
`x-pack/platform/plugins/shared/streams/scripts/extract_otel_mappings.js`
## Testing
The only way to test this logic is to setup an AI connector and use the
"Generate patterns" button.
To setup EIS see email from Dario title "AI for everyone":
1. Run command - `VAULT_ADDR={...} vault login -method oidc`
2. Run command - `node scripts/eis.js` (This will output the config for
the connector which needs to be pasted to kibana.dev.yml)
3. Run Elasticsearch - `yarn es snapshot --license trial -E
xpack.inference.elastic.url=http://localhost:8443`
4. Start Kibana as usual
5. The connector should be visible in the connectors list in Kibana and
in the AI Assistant
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Joe Reuter <johannes.reuter@elastic.co>
This PR updates the ES|QL grammars (lexer and parser) to match the
latest version in Elasticsearch.
- The `RERANK` command is in development by Elasticsearch, as such, this
PR comments out and disables `RERANK` tests where necessary to get the
lates grammar changes merged. Kibana will add `RERANK` command support
once it is ready in Elasticsearch.
---------
Co-authored-by: vadimkibana <vadimkibana@gmail.com>
## Summary
Follow up to #210579. Close#193683.
This PR removes the deprecated `visualization:useLegacyTimeAxis`
advanced setting and all related code. This setting was previously used
to enable legacy time axis behavior for charts in Lens, Discover,
Visualize and TSVB, but has been deprecated and is no longer needed. All
usage of the setting was already removed in #210579, this PR just cleans
up the advanced setting itself.
## Changes
- Removed the `visualization:useLegacyTimeAxis` UI setting registration
from the charts plugin
- Removed `LEGACY_TIME_AXIS` constant export from charts common module
- Cleaned up related TypeScript configuration dependencies
- Removed telemetry schema entries for the deprecated setting
- Removed translation strings for the deprecated setting
- Adds a saved object migration to remove the setting (see [advanced
settings
migrations](https://docs.elastic.dev/kibana-dev-docs/tutorials/advanced-settings#registering-migrations))
### Identify risks
- Identified that a saved object migration needs to be added for the
removal of the setting. See this
[PR](https://github.com/elastic/kibana/pull/157699) for reference.
- Advanced settings [can be
overridden](https://www.elastic.co/docs/extend/kibana/ui-settings-service#uisettings-overrides)
in `kibana.yml`. If the setting is present in `kibana.yml` and we remove
support for it, if verification of the file was too strict, it would
fail to start Kibana. I verified that Kibana still starts when support
for the advanced setting was removed but it's still present in
`kibana.yml`.
## Release note
Removal of the deprecated `visualization:useLegacyTimeAxis` advanced
setting.
Closes https://github.com/elastic/kibana/issues/179173.
## Rule type
A dedicated stack rule type was created `datasetQuality.degradedDocs`.
<img width="1759" alt="image"
src="https://github.com/user-attachments/assets/5004a08d-6f12-4f5e-b27f-5f4db242dcf0"
/>
<img width="2318" alt="image"
src="https://github.com/user-attachments/assets/f8b2664a-f1c6-48c5-a617-c6f1b79bf0f7"
/>
This new rule is aggregated by default using `_index` and could be
further configured by the user (e.g. user can also aggregate by
`cloud.provider`).
A new rule type was needed to be created since there is no actual way to
aggregate all documents in a dataStream if we use a DataView like
`logs-*-*`. Inside datasStream documents there is no indication about
the dataStream where they belong to, instead we just have `_index` which
contains backingIndexName instead of actual index.
It's important to note, that this rule type is also visible from
`Observability > Alerts`, which is useful specially for serverless.
https://github.com/user-attachments/assets/000aee51-4895-4f4c-9484-924ace4325c5
## Role Based Access-Control (RBAC)
RBAC for dataset quality alerts is defined within dataQuality kibana
feature.
We have three privileges defined:
1. `all`: This privilege now contains a subFeature `manage_rules` that
will allow for more granularity on alerting level. It's by default
assigned to `all` but can be disabled.
2. `read`: This privilege is only related to serverless (when we don't
have yet custom roles).
https://github.com/user-attachments/assets/70ed5bde-bf45-4024-b448-228799fcaf71
3. `none`: This privilege is only relevant for stateful (in serverless
we don't have custom roles).
## 🎥 Demo
### Serverless
#### `all` privileges
https://github.com/user-attachments/assets/8dad6e30-a261-4a69-979f-6dfc2a41c888
#### `read` privileges
https://github.com/user-attachments/assets/e1cb108d-22a0-4e7f-b252-9cc12d1e9d65
### Stateful
#### `all` privileges
https://github.com/user-attachments/assets/d96f3b70-35b2-466b-aa59-a07190d24d93
#### `all` privileges with subFeature disabled
https://github.com/user-attachments/assets/808ab811-9320-43e4-b2a6-06d530a78b82
#### `none` privileges (Stateful)
https://github.com/user-attachments/assets/18f2a2d6-d825-4713-acea-0d72f451e9ab
## How to test?
1. run synthrace scenario `degraded_logs` in live mode
```
node scripts/synthtrace degraded_logs --live
```
2. Open dataset quality page (/app/management/data/data_quality)
3. Select `synth.3` dataset
(/app/management/data/data_quality/details?pageState=(dataStream:logs-synth.3-default)
4. Click on `Actions` and then select `Create rule`
5. Fill out the alert form
6. Go to `Observability > Alerts` or `Stack management > Alerts`
(/app/observability/alerts)
## Release note
Adds the Create alert rule action to dataset quality page and dataset
quality details. This allows you to generate an alert when the
percentage of degraded docs on the chart crosses a certain threshold.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@elastic.co>
Co-authored-by: Faisal Kanout <faisal.kanout@elastic.co>
