Related to #183220
## Summary
This PR extracts `getEcsGroups` to a package to save ECS groups in the
Alert As Data (AAD) document for the metric threshold rule.
### 🧪 How to test
- Create a metric threshold rule with multiple groups (both ECS and
non-ECS fields)
- Check the related AAD document; you should be able to see the ECS
fields at the root level and not see non-ECS fields there
- Check the same information for the recovered alerts
- Rules without group by should work as before
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
**Resolves: https://github.com/elastic/kibana/issues/180153**
**Resolves: https://github.com/elastic/kibana/issues/188277**
## Summary
- Extend the POST /upgrade/_review API endpoint's contract and
functionality
- Changes `has_conflict` property within each rule field's
`ThreeWayDiff` from `boolean` to `enum` with possible values:
- `NONE`: no conflicts in three way diff
- `SOLVABLE`: conflict detected but was successfully resolved by our
algorithms
- `NON_SOLVABLE`: conflict detected and could not be resolved by our
algorithms.
- Adds `has_base_version` boolean field within each field diff
calculation. Has values:
- true: the base version of the field was found and is either defined or
undefined
- false: the base version of the field was not found
- The possible values for `has_conflict` for each concrete diff
algorithm are:
- **single line strings**: `NO`, `NON_SOLVABLE`
- **multi line strings**: `NO`, `SOLVABLE`, `NON_SOLVABLE`
- **numbers**: `NO`, `NON_SOLVABLE`
- **array of scalar values**: `NO`, `SOLVABLE`
- [ ] Adds new logic to handle
https://github.com/elastic/kibana/pull/186435#issuecomment-2178484224
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
Closes#188757
## Summary
This PR adds an endpoint to get the hosts (monitored by the system
integration) count. Currently, it supports only hosts but it can be
extended to other asset types (if/when needed). So the endpoint ( **POST
/api/infra/{assetType}/count** ) supports only 'host' as `{assetType}`.
⚠️ This PR adds only the endpoint - it is not used yet! To avoid having
different host counts and results shown on the UI this PR is not
updating the hook responsible for the request because currently the
hosts shown in the table are not filtered by the system integration
(showing the filtered result of this endpoint can lead to
inconsistencies between the count and the results shown in the table)
Once [#188756](https://github.com/elastic/kibana/issues/188756) and
[#188752](https://github.com/elastic/kibana/issues/188752) are done we
can use this endpoint.
## Testing
It can't be tested in the UI so we can test it:
<details>
<summary>Using curl:</summary>
```bash
curl --location -u elastic:changeme 'http://0.0.0.0:5601/ftw/api/infra/host/count' \
--header 'kbn-xsrf: xxxx' \
--header 'Content-Type: application/json' \
--data '{
"query": {
"bool": {
"must": [],
"filter": [],
"should": [],
"must_not": []
}
},
"from": "2024-07-23T11:34:11.640Z",
"to": "2024-07-23T11:49:11.640Z",
"sourceId": "default"
}'
```
</details>
In case of testing with oblt replace the `elastic:changeme` with your
user and password
## Summary
Implements a new `useSearchAlertsQuery` hook based on TanStack Query to
replace the `useFetchAlerts` hook, following [this organizational
logic](https://github.com/elastic/kibana/issues/186448#issuecomment-2228853337).
This PR focuses mainly on the fetching logic itself, leaving the
surrounding API surface mostly unchanged since it will be likely
addressed in subsequent PRs.
## To verify
1. Create rules that fire alerts in different solutions
2. Check that the alerts table usages work correctly ({O11y, Security,
Stack} alerts and rule details pages, ...)
1. Check that the alerts displayed in the table are coherent with the
solution, KQL query, time filter, pagination
2. Check that pagination changes are reflected in the table
3. Check that changing the query when in pages > 0 resets the pagination
to the first page
Closes point 1 of https://github.com/elastic/kibana/issues/186448
Should fix https://github.com/elastic/kibana/issues/171738
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
**Addresses**: https://github.com/elastic/kibana/issues/184428
## Summary
This PR adds scripts for automatic bundling of Timeline API OpenAPI
specs as a part of PR pipeline. Corresponding result bundles are
automatically committed to the Security Solution plugin
`x-pack/plugins/security_solution` in the `docs/openapi/ess/` and
`docs/openapi/serverless` folders (similar to
https://github.com/elastic/kibana/pull/186384).
---------
Co-authored-by: Jan Monschke <jan.monschke@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Closes: https://github.com/elastic/observability-dev/issues/3371
## Description
The Obs Alert Rule Detail view has a card that is clickable with a
focusable element inside it. This is a confusing paradigm and prevents
keyboard users from filtering by all alerts because it's not focusable.
It would be better to make the two alert number widgets the focusable
elements. Screenshot attached below.
PR is based on the following comment posted by @1Copenut in
https://github.com/elastic/observability-dev/issues/3371#issuecomment-2129446431_
> @alexwizp Agreed, panels should not be focusable. The highlighted
panel is clickable, and that was unexpected. I could click the entire
panel, and click the "1 Active now" text to filter by all alerts or
active alerts in the table below.
>
> It would be better to have the "All alerts" text be clickable and
focusable, and keep the "1 Active now" clickable and focusable. That way
the two text blocks have the interactive behavior, while the panel
(card) is just a container.
### Steps to recreate
1. Open the [Obs
Alerts](https://keepserverless-qa-oblt-b4ba07.kb.eu-west-1.aws.qa.elastic.cloud/app/observability/alerts)
table
2. Click the "Manage Rules" link
3. Create a new rule and verify it appears in the Rules table
4. Click on the rule name to load the Rule Detail view
6. Verify the `1 Active Now`
### What was done?:
1. The click event was **REMOVED** from the panel and has been moved to
`All alerts.`
2. `aria-describedby` attributes were added for `AllAlertCounts` and
`ActiveAlertCounts`
3. `h3` attributes were replaced to `EuiTitle` in `AllAlertCounts` and
`ActiveAlertCounts`
Closes: https://github.com/elastic/observability-dev/issues/3619
## Description
Observability has a number of description lists that are used to
identify important key-value pairs for users. Visually these look like a
single list of pairs, but they are coded as multiple single item lists.
This is potentially confusing to screen reader users because these data
points are related and should be in a single, consolidated list.
## What was changed?:
1. `RuleDefinition` has been updated to use `EuiDescriptionList`.
2. **The UI was affected.** In the main branch, the `Actions` item is
displayed in a separate column. With `EuiDescriptionList`, it's
displayed with other items.
## Screen:
<img width="1274" alt="image"
src="https://github.com/user-attachments/assets/4634d44f-2c0b-423f-82a2-5aa04477e142">
## Summary
Relates to https://github.com/elastic/kibana/issues/185040
This PR makes the following Fleet agents API space aware:
* `PUT /agents/{agentId}`
* `DELETE /agents/{agentId}`
* `POST /agents/bulk_update_agent_tags`
Actions created from `POST /agents/bulk_update_agent_tags` have the
`namespaces` property populated with the current space.
I am opening this PR with a few endpoints to get early feedback and make
this more agile. Other endpoints will be implemented in a followup PR.
### Testing
1. Enroll an agent in the default space.
2. Create a custom space and enroll an agent in it.
3. From the default space, test the `PUT /agents/{agentId}` and `DELETE
/agents/{agentId}` endpoints and check that the request fails for the
agent in the custom space.
4. Same test from the custom space.
5. From the default space, test the `POST
/agents/bulk_update_agent_tags` with all agents ids and check that only
the agents in the default space get updated.
6. Same test from the custom space.
7. Review the actions created from the bulk tag updates (the easiest way
is `GET .fleet-actions/_search`) and ensure the `namespaces` property is
correct.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Resolves#189048. This PR fixes the content type and file name for the
agent policy that is downloaded for standalone agents, following
https://github.com/elastic/kibana/pull/187133.
Also adjusts some copy casing.
**Addresses**: https://github.com/elastic/kibana/issues/184428
## Summary
This PR adds scripts for automatic bundling of Osquery API OpenAPI specs as a part of PR pipeline. Corresponding result bundles are automatically committed to the Osquery plugin `x-pack/plugins/osquery` in the `docs/openapi/ess` and `docs/openapi/serverless` folders (similar to https://github.com/elastic/kibana/pull/186384).
**Addresses**: https://github.com/elastic/kibana/issues/184428
## Summary
This PR adds scripts for automatic bundling of AI Assistant API OpenAPI specs as a part of PR pipeline. Corresponding result bundles are automatically committed to the Assistant Common package `x-pack/packages/kbn-elastic-assistant-common` in the `docs/openapi/ess/` and `docs/openapi/serverless` folders (similar to https://github.com/elastic/kibana/pull/186384).
The knowledge team is a codeowner of the other data access plugins and
should be on Logs.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Resolves https://github.com/elastic/kibana/issues/185043
## Summary
### Task types can define a `cost` associated with running it
- Optional definition that defaults to `Normal` cost
### New `xpack.task_manager.capacity` setting
- Previous `xpack.task_manager.max_workers` setting is deprecated,
changed to optional, and a warning will be logged if used
- New optional `xpack.task_manager.capacity` setting is added. This
represents the number of normal cost tasks that can be run at one time.
- When `xpack.task_manager.max_workers` is defined and
`xpack.task_manager.capacity` is not defined, a deprecation warning is
logged and the value for max workers will be used as the capacity value.
- When `xpack.task_manager.capacity` is defined and
`xpack.task_manager.max_workers` is not defined, the capacity value will
be used. For the `default` claiming strategy, this capacity value will
be used as the `max_workers` value
- When both values are set, a warning will be logged and the value for
`xpack.task_manager.capacity` will be used
- When neither value is set, the `DEFAULT_CAPACITY` value will be used.
### Updates to `TaskPool` class
- Moves the logic to determine used and available capacity so that we
can switch between capacity calculators based on claim strategy. For the
`default` claim strategy, the capacity will be in units of workers. For
the `mget` claim strategy, the capacity will be in units of task cost.
### Updates to `mget` task claimer
- Updated `taskStore.fetch` call to take a new parameter that will
return a slimmer task document that excludes that task state and task
params. This will improve the I/O efficiency of returning up to 400 task
docs in one query
- Applies capacity constraint to the candidate tasks.
- Bulk gets the full task documents for the tasks we have capacity for
in order to update them to `claiming` status. Uses the
`SavedObjectsClient.bulkGet` which uses an `mget` under the hood.
### Updates the monitoring stats
- Emitting capacity config value and also capacity as translated into
workers and cost.
- Added total cost of running and overdue tasks to the health report
## Tasks for followup issues
- Update mget functional tests to include tasks with different costs. -
https://github.com/elastic/kibana/issues/189111
- Update cost of indicator match rule to be Extra Large -
https://github.com/elastic/kibana/issues/189112
- Set `xpack.task_manager.capacity` on ECH based on the node size -
https://github.com/elastic/kibana/pull/189117
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Resolves https://github.com/elastic/kibana/issues/188804
Allows retest on failure to be configured in the monitor creation UI.
https://www.loom.com/share/2302ccfd150b4668a61966be165478f8
### Testing
1. Create a monitor and toggle retest on failure off.
2. Navigate to edit that monitor. Ensure the setting remains off.
### Release note
Fixes a bug where retest on failure was not able to be turned off when
creating a monitor in the Synthetics app. Retest on failure can now be
turned off.
## Summary
PR adds logic to background task to check and complete `processes`
response actions for SentinelOne. Changes include:
- Added logic, invoked by background task, to check and complete pending
`processes` response actions for SentinelOne
- UI: the Response console was adjusted for `processes` command to show
the download link to the file that contains the output of the
`processes` for sentinelone
- ℹ️ SentinelOne does not return the actual results for display, but
rather provides a `.zip` file that contains the output in `.csv` format
- Fixed a UI issue with the file download component where every time the
browser window was focus on, the component would show a loading
animation
- Updated the SentinelOne dev script so that the integration policy
created uses `30s` for the data pulling interval
## Summary
- addresses https://github.com/elastic/security-team/issues/9228
- introduces new Kibana advanced settings option
`securitySolution:excludedDataTiersForRuleExecution`, that allows to
exclude cold and frozen data tiers from search during rule execution
- users would be able to add `data_cold` or/and `data_frozen` tiers
- **ES|QL** rule does not support this feature:
https://github.com/elastic/elasticsearch/issues/108264
- **Machine learning** rule does not support this feature
- Advanced setting available only for ESS
### UI
<img width="2300" alt="Screenshot 2024-07-04 at 17 31 34"
src="39beeda3-8030-4943-959c-53eb064fe5ae">
### Demo
1. Checking there are 3M+ documents in cold data tier of `test-frozen`
index
2. When rule executes, it generates alerts.
3. Checking kibana ancestor index of generated alert - it's
`restored-test-frozen-000001`, which confirms alert was created from a
document in cold tier
4. In advanced settings exlcude `data_cold` tier
5. Execute rule again, observe no alerts were created
c8b2f612-628a-452d-98e5-555c2e89d957
### How to test
Create a deployment with cold and frozen data tiers and use following
commands to create index and ILM
<details>
<summary>Data tiers commands</summary>
```JSON
PUT /_cluster/settings
{
"persistent": {
"indices.lifecycle.poll_interval": "1m"
}
}
PUT /_ilm/policy/filtering_data_tiers
{
"policy": {
"phases": {
"frozen": {
"min_age": "10m",
"actions": {
"searchable_snapshot": {
"snapshot_repository": "found-snapshots",
"force_merge_index": true
}
}
},
"cold": {
"min_age": "1m",
"actions": {
"searchable_snapshot": {
"snapshot_repository": "found-snapshots",
"force_merge_index": true
},
"set_priority": {
"priority": 0
}
}
},
"hot": {
"min_age": "0ms",
"actions": {
"set_priority": {
"priority": 100
}
}
}
}
}
}
PUT /_index_template/filtering_data_tiers_template
{
"index_patterns": [
"filtering_data_tiers*"
],
"template": {
"settings": {
"index.lifecycle.name": "filtering_data_tiers",
"index.lifecycle.rollover_alias": "test-filtering_data_tiers"
},
"mappings": {
"_meta": {
"version": "1.6.0"
},
"properties": {
"@timestamp": {
"type": "date"
},
"host": {
"properties": {
"name": {
"type": "keyword",
"ignore_above": 1024
}
}
}
}
}
}
}
PUT /filtering_data_tiers-000001
{
"aliases": {
"filtering_data_tiers": {
"is_write_index": true
}
}
}
POST filtering_data_tiers/_doc
{
"@timestamp": "2024-07-08T17:00:01.000Z",
"host.name": "test-0"
}
```
</details>
**OR**
reach out to @vitaliidm to get access to already existing deployment/pR
deployment, where `test-frozen` index has cold and frozen nodes and ILM
policy that move any data to a tier according to config.
Check number of documents in tier by
```JSON
GET test-frozen/_count
{
"query": {
"bool": {
"must": {
"terms": {
"_tier": ["data_cold"]
}
}
}
}
}
```
Create rule of supported type and query that index
### Checklist
- [x] Functional changes are covered with a test plan and automated
tests.
- https://github.com/elastic/security-team/pull/9896
- [x] Comprehensive manual testing is done by two engineers: the PR
author and one of the PR reviewers. Changes are tested in both ESS and
Serverless.
- [x] Functional changes are communicated to the Docs team. A ticket or
PR is opened in https://github.com/elastic/security-docs. The following
information is included: any feature flags used, affected environments
(Serverless, ESS, or both).
- https://github.com/elastic/security-docs/issues/5483
