## Summary
Multi space monitors !!
Fixes https://github.com/elastic/kibana/issues/164294
User will be able to choose in which space monitors will be available !!
<img width="1728" alt="image"
src="https://github.com/user-attachments/assets/f01ac226-ed54-4e96-b6f4-27f0134a9be5"
/>
### Technical
This is being done by registering another saved object type and for
existing monitors it will continue to work as right now but for newly
created monitors user will have ability to specify spaces or choose
multiple spaces or all.
### Testing
1. Create few monitors before this PR in multiple spaces
2. Create multiple monitors in multiple spaces after this PR
3. Make sure filtering, editing and deleting, creating works as expected
on both set of monitors
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/ingest-dev/issues/5444
* Add `enablePackageRollback` feature flag
* Save package policy previous revision on package upgrade
* Add `latest_revision` boolean property to
`ingest-package-policies/fleet-package-policies` saved object type
* Package policy SO are created with `latest_revision: true`
* When a package policy is updated with a new package version, the
previous SO is saved to ES with id `{id}:prev` and `latest_revision:
false`
* Backfill existing SO with `latest_revision: true`
* GET logic filters for `latest_revision: true`
* Save package previous version
* Add `previous_version` property to `epm-packages` saved object type
* When a package is upgraded to a new version, set `previous_version`
### Testing
* Install an integration on an outdated version (edit the version in the
URL and add the integration).
* Check the package policy SO: it should have been created with
`latest_revision: true`.
* Check the package SO: the `previous_version` property should not be
set.
* Upgrade the integration and upgrade package policies.
* Check the package policy SO: there should now be 2 SO for this package
policy:
* The updated one with `latest_revision: true` and policy id
* The previous one with `latest_revision: false` and `{policy_id}:prev`
* Check the package SO: the `previous_version` property should be set to
the old version
Note: it seems Fleet only allows upgrading packages to the latest
version (please correct me if that's wrong); for testing two consecutive
updates (e.g. check that only the most recent revision is saved), it
might be necessary to run a custom EPR.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
Risk of bad requests across Fleet wherever packages or package policies
are queried.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
fixes https://github.com/elastic/kibana/issues/211540
User will be able to choose maintenance window in the form
<img width="1723" alt="image"
src="https://github.com/user-attachments/assets/c4d75aff-687f-40d3-a614-160e99ce9ac2"
/>
A callout will be displayed on the form
<img width="1728" alt="image"
src="https://github.com/user-attachments/assets/124727bd-0bb6-4934-9406-a36c3584670a"
/>
### Task manager
When changes are made to maintenance windows, those are sync via task
manager to private location monitors, public location monitors are
automatically synced as well in already existing task.
### Testing
Create a maintenance window in stack management UI, apply it to monitor,
make sure, it never runs during maintenance window.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
PR adds migration of Response Actions and Artifacts when the space
awareness feature flag is enabled. Changes include:
- New saved object type (`security:reference-data`) to keep internal
data/information used by security solution. Currently being used to keep
migration state for both the Artifacts and Response actions migrations
- Two records will be populated in this new saved object related to the
migration being introduced here:
- ID: `SPACE-AWARENESS-ARTIFACT-MIGRATION` - keeps state information
about migration of Artifacts (see below for example)
- ID: `SPACE-AWARENESS-RESPONSE-ACTIONS-MIGRATION` - keeps state
information about migration of response actions (see below for example)
- NOTE: my goal was to also introduce a `client` for working with
internal reference data, but due to timing constraints, that will be
done at a later time
- Adds migration for Endpoint artifacts (Trusted applications, Event
filters, Blocklists, Host isolation exceptions and Endpoint exceptions)
to add the `spaceOwnerId:default` to all artifact records
- Adds migration for Response Action Requests (both Elastic Defend and
external EDRs) to populate `originSpaceId` as well `agent.policy.*`
fields.
- NOTE: when populating the `agent.policy.*` data, it may be possible
that the agent that the response action was sent to is no longer
available (ex. unenrolled). In such cases, the migration will populate
any field that it is not able to retrieve information for with the text
`MIGRATION:NOT-FOUND`
> [!IMPORTANT]
> The migration currently assumes that the `9.1.0` endpoint package is
already installed, which will not be the case with a real migration. To
test this PR, you should start a 9.1 stack with the feature flags
disabled... Install the `9.1.0` endpoint package, and then enable the
feature flags.
> We will be working on a solution handling this data condition in a
subsequent PR
## Summary
This PR introduces a new Saved Object mapping for Monitoring Entity
Source Configurations to support Privileged User Monitoring.
### ✅ Implemented
* Saved object mapping schema aligned with the [current
spec](https://docs.google.com/document/d/1-c6A82p2CCjYz8Mb6tL6R7BXXemQ4mpWZliKqki0SWo/edit?tab=t.0#heading=h.4axxr154e1lc)
* DescriptorClient to handle saved object methods
* OpenAPI schema and generated types
* Implemented Data Client - SO operations exposed via GET and POST API
* Created route and registered with privmonRoutes
* Data Client Unit Testing
* Rename typename of "monitoring_entity_source-sync" ->
"entity-analytics-monitoring-entity-source"
[Figma
](https://www.figma.com/board/yBr1pBDGu4JqNxb5ZrULtk/MonEntitySourceSyncTask?node-id=0-1&p=f&t=4hQaGsdvOpghi4QS-0)-
currently working in the pink box
## Testing Steps:
1. privilegeMonitoringEnabled: true - set this flag
2. From Dev tools - create a saved object via POST (below) and view this
saved object via GET (also below):
```
POST kbn:/api/entity_analytics/monitoring/entity_source
{
"type": "some-type-here",
"name": "name-here",
"managed": false,
"indexPattern": "logs-*",
"enabled": true,
"integrationName": "Okta",
"matchers": [
{
"fields": ["user.role"],
"values": ["admin"]
}
],
"filter": {}
}
```
**Expected output:**
```
{
"type": "some-type-here",
"name": "name-here",
"managed": false,
"indexPattern": "logs-*",
"enabled": true,
"integrationName": "Okta",
"matchers": [
{
"fields": [
"user.role"
],
"values": [
"admin"
]
}
],
"filter": {}
}
```
GET Example
```
GET kbn:/api/entity_analytics/monitoring/entity_source
```
Output for get, expected to be the same as that for POST.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
## Summary
This is the first PR related to
https://github.com/elastic/kibana/issues/212570. (more up-to-date
information in the [design
doc](https://docs.google.com/document/d/1DZKTPl7UryYjpjVMNhIYbE82OADVOg93-d02f0ZQtUI/edit?tab=t.0))
It only introduces the `incremental_id` field to the cases object and
introduces the new "id incrementer" saved object. Tests and migrations
have been changed accordingly and the `incremental_id` field is removed
from cases import/export.
The motivation behind releasing these changes first is so that
serverless deployments will know about the new field and SO once the
other changes are coming.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Closes https://github.com/elastic/kibana/issues/1547
Closes https://github.com/elastic/kibana/issues/190342
Closes https://github.com/elastic/kibana/issues/197716
## Summary
This PR adds the ability for collapsible sections to be created and
managed on Dashboards.
https://github.com/user-attachments/assets/c5c046d0-58f1-45e1-88b3-33421f3ec002
> [!NOTE]
> Most of the work for developing collapsible sections occurred in PRs
contained to the `kbn-grid-layout` package (see [this meta
issue](https://github.com/elastic/kibana/issues/190342) to track this
work) - this PR simply makes them available on Dashboards by adding them
as a widget that can be added through the "Add panel" menu. As a result
of this, most work is contained in the Dashboard plugin - changes made
to the `kbn-grid-layout` package only include adding IDs for additional
tests that were added for the Dashboard integration.
### Technical Details
#### Content Management Schema
The content management schema allows for panels and sections to be mixed
within the single `panels` key for a dashboard **without** worrying
about section IDs; for example:
```
{
"panels": [
{
// this is a simplified panel
"gridData": {
"x": 0,
"y": 0,
"w": 12,
"h": 8,
},
"panelConfig": { ... },
},
{
// this is a section
"gridData": {
"y": 9,
},
"collapsed": false,
"title": "Section title",
"panels": [
{
// this is a simplified panel
"gridData": {
"x": 0,
"y": 0,
"w": 24,
"h": 16,
},
"panelConfig": { ... },
},
],
},
]
}
```
#### Saved Object Schema
The dashboard saved object schema, on the other hand, separates out
sections and panels under different keys - this is because, while we are
stuck with panels being stored as `panelJSON`, I didn't want to add much
to this. So, under grid data for each panel, they have an optional
`sectionId` which then links to a section in the `sections` array in the
saved object:
```
{
"panelsJSON": "<...> \"gridData\":{\"i\":\"panelId\",\"y\":0,\"x\":0,\"w\":12,\"h\":8,\"sectionId\":\"someSectionId\"} <...>"
"sections": [
{
"collapsed": false,
"title": "Section title",
"gridData": {
"i": "someSectionId",
"y": 8.
}
}
],
}
```
This allows sections to be serialized **without** being stringified.
This storage also matches how we store this data in runtime using
`layout`.
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Release note
Adds collapsible sections to Dashboard, which allow panels to grouped
into sections that will not load their contents when their assigned
section is collapsed.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR aims to answer questions about the implementation details for
https://github.com/elastic/kibana-team/issues/1328
We'd like to trigger an intercept dialog to users at specific time
intervals that vary and depend on different parameters, see the PRD
linked in the aforementioned issue.
This MVP takes an approach such that the constraints listed below are
resolved;
- Have the intercept be specific to a single user
- Define a strategy to configure triggers that can vary for the
intercept that's not dependent on the client
How does this work?
- A generic plugin has been created which when declared as dependency
can be used to register and schedule an intercept that should be
presented to the user, this plugin is what's been used for the product
intercept dialog.
- To handle rendering the UI component that gets displayed to the users,
in this PR we hook into core's existing notification system, through the
notification coordinator system that's been created so that we don't
have a situation where an intercept is being displayed whilst a user
might have a toast being rendered to them, if there's an intercept to be
displayed said intercept would not be displayed till the user is
completely done interacting with the toast. [See it's implementation
details
here](src/core/packages/notifications/browser-internal/src/notification_coordinator.ts)
and
[here](x-pack/platform/plugins/private/intercepts/public/prompter/service/intercept_dialog_service.tsx)
for how it's integrated.
- The plugin provides some bootstrap data through an endpoint that every
user calls on page load, ideally this would need to happen just the once
on page load, we then compute when a user should see a trigger
leveraging the bootstrap data provided for the particular registered
trigger in question, the returned data returns the following data as
seen below;
<img width="476" alt="Screenshot 2025-03-27 at 18 01 12"
src="https://github.com/user-attachments/assets/c747b6c8-70d0-4305-b555-ec9998b180c1"
/>
Given we have this data we might then have a flow for triggering the
intercept on the client based of the bootstrap data like so;
```mermaid
flowchart TD
A[State Bootstrap] --> B{Has data?}
B -->|No| C(Do Nothing)
B -->|Yes| D{Does computed runs since trigger registration match stored
user trigger run feedback value?}
D -->|No| E[setup timer to display an intercept in that time]
E -->|on completion| G[setup interval to display intercept in the
future]
D -->|Yes| G
G -->|repeat| G
```
reloading the page restarts the entire process.
## Telemetry
The intercept component provided by defaults records telemetry for
intercept registration, acknowledgement (differentiated into dismissal
and completion), alongside registration overload.
## Visuals
<!--
https://github.com/user-attachments/assets/b39a506c-a119-40e8-9152-258d78691f28
-->
<!--
https://github.com/user-attachments/assets/f564b4bc-9ad9-4e19-8158-6e154ef52fc2
-->
<img width="738" alt="Screenshot 2025-05-07 at 19 41 23"
src="https://github.com/user-attachments/assets/902c6d0b-9299-44bd-8808-4ad97227d0da"
/>
## Testing this PR
- Pull this branch to your machine
- Add the following to your `kibana.dev.yml`, to enable the intercept to
run and be visible
```yml
xpack.intercepts.enabled: true
xpack.product_intercept.enabled: true
xpack.product_intercept.interval: '30s'
```
<!--
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.
Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.
- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...
-->
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Part of https://github.com/elastic/kibana-team/issues/1503
This PR is mostly about moving tests from `x-pack/test` to
`x-pack/platform/test`
Before:
```
x-pack/test/
| - api_integration/apis/management/
| - api_integration/apis/monitoring/
| - api_integration/apis/osquery/
| - automatic_import_api_integration
| - monitoring_api_integration/
| - saved_object_api_integration/
```
After:
```
x-pack/platform/test/
| - api_integration/apis/management/
| - api_integration/apis/monitoring/
| - api_integration/apis/osquery/
| - automatic_import_api_integration
| - monitoring_api_integration/
| - saved_object_api_integration/
```
Additional:
- `x-pack/test/api_integration/apis/osquery/config.ts` was move to
platform stateful manifest to match `osquery` plugin (shared, platform)
- `automatic_import_api_integration` config files were moved to platform
stateful manifest to match `automatic_import` plugin (shared, platform)
- `x-pack/test/common/lib/test_data_loader.ts` is required by SO api
tests and was copied to `x-pack/platform/test/common`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Related to #213024.
This adds the necessary backend schema changes for the investigation
guide.
Note this PR depends on #216292 and should not be merged before it.
---------
Co-authored-by: Panagiota Mitsopoulou <giota85@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Panagiota Mitsopoulou <panagiota.mitsopoulou@elastic.co>
Co-authored-by: Christos Nasikas <xristosnasikas@gmail.com>
## Summary
This PR creates CRUD endpoints for search playgrounds. This will enable
us to make new pages for saved playgrounds that are shared in a space.
## Notes
Usages of `ALL_SAVED_OBJECT_INDICES` had to be updated to include
`ignore_unavailable` since a new index was added for search solution
saved objects, but these are not always registered when search plugins
are disabled. Because of this refresh and other calls using
`ALL_SAVED_OBJECT_INDICES` were failing when the new
`.kibana_search_solution` index did not exist.
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Gerard Soldevila <gerard.soldevila@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
This PR introduces the first building blocks for the [Entity Analytics
Privileged
Monitoring](https://github.com/elastic/security-team/issues/9971).
We follow the approach used in the Entity Store and add a new "Engine",
which consists of the following components:
* Public API
* INIT and HEALTH routes
* Kibana task
* Privilege Monitoring Data Client
* Engine Saved Object
* API key manager
* Related storage indices
* Feature Flag: `privilegeMonitoringEnabled` set to `false` by default.
* API integration test configuration
* only tests that the health endpoint is available
* Auditing and Telemetry
## Testing steps
1. Make sure to add `privilegeMonitoringEnabled` to your
`kibana.dev.yaml`
2. In devtools, ensure the API is working with `GET
kbn:/api/entity_analytics/monitoring/privileges/health`
3. Start the engine with: `POST
kbn:/api/entity_analytics/monitoring/engine/init`
4. Look for `DEBUG` logs mentioning the
`entity_analytics:monitoring:privileges:engine` task
---------
Co-authored-by: CAWilson94 <charlotte.wilson@elastic.co>
Co-authored-by: Charlotte Alexandra Wilson <CAWilson94@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/206556
This PR adds a setting to remote ES outputs for also uninstalling
integrations on remote clusters when integrations sync is enabled.
This new setting can be toggled in the UI with a new switch:
<img width="1728" alt="Screenshot 2025-04-09 at 11 53 43"
src="https://github.com/user-attachments/assets/34544aa9-28fd-4360-a32f-5031e3d4293f"
/>
### Testing
* Follow the steps in
https://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/shared/fleet/dev_docs/local_setup/remote_clusters_ccr.md
to set up two clusters with integrations syncing.
* Add some integrations in your main cluster and check that they are
also installed in the remote cluster.
* Disable uninstalling integrations on remote.
* Remove an integration in your main cluster and check that it is NOT
removed from the remote cluster.
* Enable uninstalling integrations on remote.
* Remove an integration in your main cluster and check that it is also
removed from the remote cluster.
* In your remote cluster, enroll an agent onto a policy that points to
at least 1 package policy of the installed integrations (cf. Docker
commands below if using dockerized fleet-server/agent).
* In your main cluster, uninstall the integration that is used by the
agent policy in the remote. This should cause the uninstall to fail into
the remote cluster.
* In your remote cluster, inspect the package SO of that integration
with `GET .kibana_ingest/_search?q=type:epm-packages`: the
`latest_uninstall_failed_attempts` field should be populated.
Docker command for running a fleet-server in your remote cluster:
```
docker run \
-e ELASTICSEARCH_HOST=http://host.docker.internal:9500 \
-e KIBANA_HOST=http://host.docker.internal:5701/<path> \
-e KIBANA_USERNAME=elastic \
-e KIBANA_PASSWORD=changeme \
-e KIBANA_FLEET_SETUP=1 \
-e FLEET_INSECURE=1 \
-e FLEET_SERVER_ENABLE=1 \
-e FLEET_SERVER_POLICY_ID=fleet-server-policy \
-p 8220:8220 \
--rm docker.elastic.co/beats/elastic-agent:9.0.0-SNAPSHOT
```
Docker command for enrolling an agent in your remote cluster:
```
docker run \
-e ELASTICSEARCH_HOST=http://host.docker.internal:9500 \
-e KIBANA_HOST=http://host.docker.internal:5701/<path> \
-e FLEET_URL=https://host.docker.internal:8220 \
-e FLEET_ENROLL=1 \
-e FLEET_ENROLLMENT_TOKEN=<token> \
-e FLEET_INSECURE=1 \
--rm docker.elastic.co/beats/elastic-agent:9.0.0-SNAPSHOT
```
### Checklist
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
This feature is currently in development and behind the
`enableSyncIntegrationsOnRemote` feature flag.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
~**DO NOT MERGE:** depends on
https://github.com/elastic/kibana/issues/213468~
This PR reintegrates the work from the `workchat_m1` branch into `main`:
- introduces a 4th solution type, `chat`, that will be used for the
*WorkChat* project type.
- edit things in various platform code to introduce/handle that new
project type
- add plugins and packages for the workchat app.
### To AppEx reviewers:
File change count is scary, but you can safely ignore anything from
`xpack/solutions/chat` (given it's solution code), and focus on your
owned changes, which are way more reasonable
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Joe McElroy <joseph.mcelroy@elastic.co>
Co-authored-by: Rodney Norris <rodney.norris@elastic.co>
Co-authored-by: Jedr Blaszyk <jedrazb@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Meghan Murphy <meghan.murphy@elastic.co>
## Summary
Closes https://github.com/elastic/ingest-dev/issues/4722
### Implementation checklist
- [x] Handle fetching agent policies and agents at scale
- [x] Only consider active agents for upgrade
- [x] Agents already on or upgrading to target version are included in
the count but not considered for upgrade
- [x] Agents stuck in updating are considered for upgrade
- [x] Bulk upgrade actions triggered by the task have an added
`isAutomatic:true` flag
- [x] Use rollout duration to spread bulk upgrade in time (1h or longer
depending on agent count)
### Testing
- This should be tested with real Elastic Agents (that will upgrade and
have `upgrade_details`).
- Edit the task interval in order to test how the task logic handles
agents already upgrading.
- Edit the agents batch size in order to test how the task logic handles
agents at scale.
- We should also check that space awareness is respected if enabled.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
Risk of incorrectly triggering agent upgrades. Probability should be
very low if the agent policy does not have `required_versions` set.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
This PR introduces support for configuring the `timestamp` field for
entity store enablement.
By default, the `timestamp` field is set to `@timestamp`, but users can
opt to use `event.ingested` or another preferred value based on their
requirements.
### Entity Store API changes
#### Entity Store enable
```
POST kbn:/api/entity_store/enable
{
"timestampField": "event.ingested"
}
```
#### Result
```
{
"engines": [
{
"status": "started",
"type": "user",
"indexPattern": "",
"filter": "",
"fieldHistoryLength": 10,
"timestampField": "event.ingested"
},
{
"status": "started",
"type": "host",
"indexPattern": "",
"filter": "",
"fieldHistoryLength": 10,
"timestampField": "event.ingested"
},
{
"status": "started",
"type": "universal",
"indexPattern": "",
"filter": "",
"fieldHistoryLength": 10,
"timestampField": "event.ingested"
}
],
"status": "running"
}
```
```
POST kbn:/api/entity_store/enable
{
}
```
#### Result
```
{
"engines": [
{
"status": "started",
"type": "host",
"indexPattern": "",
"filter": "",
"fieldHistoryLength": 10,
"timestampField": "@timestamp"
},
{
"status": "started",
"type": "user",
"indexPattern": "",
"filter": "",
"fieldHistoryLength": 10,
"timestampField": "@timestamp"
},
{
"status": "started",
"type": "universal",
"indexPattern": "",
"filter": "",
"fieldHistoryLength": 10,
"timestampField": "@timestamp"
}
],
"status": "running"
}
```
Different entity types can have distinct `timestampField` values, as
described below:
For instance, the `host` entity uses `event.ingested` as its
`timestampField`, while other entities default to the `@timestamp`
field.
```
POST kbn:/api/entity_store/engines/host/init
{
"timestampField": "event.ingested"
}
POST kbn:/api/entity_store/engines/user/init
{
}
```
#### Result
```
{
"engines": [
{
"status": "started",
"type": "universal",
"indexPattern": "",
"filter": "",
"fieldHistoryLength": 10,
"timestampField": "@timestamp"
},
{
"status": "started",
"type": "host",
"indexPattern": "",
"filter": "",
"fieldHistoryLength": 10,
"timestampField": "event.ingested"
},
{
"status": "started",
"type": "user",
"indexPattern": "",
"filter": "",
"fieldHistoryLength": 10,
"timestampField": "@timestamp"
}
],
"status": "running"
}
```
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Testing steps
- Check out this PR branch.
- Start Kibana.
- Avoid enabling the Entity Store from the UI initially. Instead, enable
it via the API by following the steps outlined below.
```
DELETE kbn:/api/entity_store/engines/user
DELETE kbn:/api/entity_store/engines/host
GET kbn:/api/entity_store/status
All engines with the same timestamp Field
POST kbn:/api/entity_store/enable
{
"timestampField": "event.ingested"
}
# All engines with the same timestamp Field
POST kbn:/api/entity_store/enable
{
}
# Different entity with different timestamp Field
POST kbn:/api/entity_store/engines/host/init
{
"timestampField": "event.ingested"
}
POST kbn:/api/entity_store/engines/user/init
{
}
```
4. Afterward, you can test the functionality through the UI. First,
clear the entity data from the UI, then start the Entity Store. Note
that the UI currently does not offer a way to configure this parameter.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Mark Hopkin <mark.hopkin@elastic.co>
## Summary
Redoing [this PR](https://github.com/elastic/kibana/pull/203184) which
had to be [reverted](https://github.com/elastic/kibana/pull/204218).
This should not be merged until [this update to the task manager v1
schema](https://github.com/elastic/kibana/pull/204413) is released.
## To verify
1. Set `xpack.task_manager.unsafe.exclude_task_types:
['ad_hoc_run-backfill', 'actions:*']` in your Kibana config.
2. Run Kibana on main and create some detection rules that run
frequently, with actions.
3. Schedule a manual run for your detection rules.
- Because of the config, the `action_task_params` SO and the
`ad_hoc_run_task_params` SO will get written but not read yet.
4. Remove the `exclude_task_types` config and "upgrade" to this PR
branch and verify that rules continue to run and that the actions are
triggered and the manual rule runs go through
5. Re-add the `exclude_task_types` config and let the rule run again to
schedule action. Schedule another manual rule run.
6. Remove the `exclude_task_types` config and "downgrade" back to main
and verify that rules continue to run, the action gets triggered and
manual rule runs go through.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This reverts commit b9bac1628b.
## Summary
Reverting to investigate `Failed to poll for work: Invalid interval
"PT1M". Intervals must be of the form {number}m. Example: 5m.` logs
which may be caused by the updated task schema
## Summary
This PR contains just the schema changes required to support backfill
actions. This is meant for an intermediate release and then the full PR:
https://github.com/elastic/kibana/pull/200784 will follow after that.
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Resolves https://github.com/elastic/ingest-dev/issues/3933. For
deployments that support agentless, integrations with agentless
deployment mode enabled will allow the status of agentless integration
policies to be tracked.
### Key technical changes
- A new field `supports_agentless` was added to package policies. This
field already exists on agent policies. When an agentless integration is
created, `supports_agentless: true` is now added to both the package
policy and its parent agent policy.
- This allows easier filtering for agentless integrations as we avoid
having to retrieve & check against every parent agent policy.
- This also means existing agentless policies do not get this new status
tracking UI, only new ones created after this change. Since agentless is
not yet GA, I think this is okay.
- `/api/fleet/agent_status/data` now takes optional query params
`pkgName` and `pkgVersion`. When both are specified, the API will check
if agent(s) have ingested data for only that package's datastreams.
## UI walkthrough
<details>
<summary>🖼️ Click to show screenshots</summary>
1. **Integration policies** page now shows two tables for integrations
meeting the above condition, one for agentless policies and one for
agent-based policies:
2. Clicking the status badge in the agentless policies table opens a
flyout with two steps: confirm agentless enrollment and confirm incoming
data:
3. Confirm agentless enrollment polls for an agent enrolled into that
integration policy's agent policy. If that agent is reporting an
unhealthy status, the integration component UI is shown. This UI is the
same one used on Fleet > Agents > Agent details page and shows all
components reported by that agent:
4. Once a healthy agentless enrollment is established, confirm incoming
data starts polling for data for that integration ingested by that agent
ID in the past 5 minutes:
5. If data could not be retrieved in 5 minutes, an error message shows
while polling continues in the background:
6. If data is retrieved, a success message is shown:
</details>
## Testing
Easiest way to test is use the Cloud deployment from this PR. Enable
Beta integrations and navigate to CSPM. Add a CSPM integration using
`Agentless` setup technology. Then you can track the status of the
agentless deployment on the Integrations policies tab.
For local testing, the following is required to simulate agentless
agent:
1. Add the following to kibana.dev.yml:
```
xpack.cloud.id: 'anything-to-pass-cloud-validation-checks'
xpack.fleet.agentless.enabled: true
xpack.fleet.agentless.api.url: 'https://localhost:8443'
xpack.fleet.agentless.api.tls.certificate: './config/certs/ess-client.crt'
xpack.fleet.agentless.api.tls.key: './config/certs/ess-client.key'
xpack.fleet.agentless.api.tls.ca: './config/certs/ca.crt'
```
2. Apply [this
patch](https://gist.github.com/jen-huang/dfc3e02ceb63976ad54bd1f50c524cb4)
to prevent attempt to create agentless pod
3. Enroll a Fleet Server as usual
4. Enable Beta integrations and navigate to CSPM. Add a CSPM integration
using `Agentless` setup technology.
5. Enroll a normal Elastic Agent to the agent policy for that CSPM
integration by using the token from Enrollment tokens
## To-do
- [x] API tests
- [x] Unit UI tests
- [x] Manual Cloud tests
- [x] File docs request
- https://github.com/elastic/ingest-docs/issues/1466
- [ ] Update troubleshooting guide link once available
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/193352
Update:
Using a new SO field `bump_agent_policy_revision` in package policy type
to mark package policies for update, this will trigger an agent policy
revision bump.
The feature supports both legacy and new package policy SO types, and
queries policies from all spaces.
To test, add a model version change to the package policy type and save.
After Fleet setup is run, the agent policies using the package policies
should be bumped and deployed.
The same effect can be achieved by manually updating a package policy SO
and loading Fleet UI to trigger setup.
```
'2': {
changes: [
{
type: 'data_backfill',
backfillFn: (doc) => {
return { attributes: { ...doc.attributes, bump_agent_policy_revision: true } };
},
},
],
},
curl -sk -XPOST --user fleet_superuser:password -H 'content-type:application/json' \ -H'x-elastic-product-origin:fleet' \
http://localhost:9200/.kibana_ingest/_update_by_query -d '
{ "query": {
"match": {
"type": "fleet-package-policies"
}
},"script": {
"source": "ctx._source[\"fleet-package-policies\"].bump_agent_policy_revision = true",
"lang": "painless"
}
}'
```
```
[2024-11-20T14:40:30.064+01:00][INFO ][plugins.fleet] Found 1 package policies that need agent policy revision bump
[2024-11-20T14:40:31.933+01:00][DEBUG][plugins.fleet] Updated 1 package policies in space space1 in 1869ms, bump 1 agent policies
[2024-11-20T14:40:35.056+01:00][DEBUG][plugins.fleet] Deploying 1 policies
[2024-11-20T14:40:35.493+01:00][DEBUG][plugins.fleet] Deploying policies: 7f108cf2-4cf0-4a11-8df4-fc69d00a3484:10
```
TODO:
- the same flag has to be added on agent policy and output types, and
the task extended to update them
- I plan to do this in another pr, so that this doesn't become too big
- add integration test if possible
### Scale testing
Tested with 500 agent policies split to 2 spaces, 1 integration per
policy and bumping the flag in a new saved object model version, the
bump task took about 6s.
The deploy policies step is async, took about 30s.
```
[2024-11-20T15:53:55.628+01:00][INFO ][plugins.fleet] Found 501 package policies that need agent policy revision bump
[2024-11-20T15:53:57.881+01:00][DEBUG][plugins.fleet] Updated 250 package policies in space space1 in 2253ms, bump 250 agent policies
[2024-11-20T15:53:59.926+01:00][DEBUG][plugins.fleet] Updated 251 package policies in space default in 4298ms, bump 251 agent policies
[2024-11-20T15:54:01.186+01:00][DEBUG][plugins.fleet] Deploying 250 policies
[2024-11-20T15:54:29.989+01:00][DEBUG][plugins.fleet] Deploying policies: test-policy-space1-1:4, ...
[2024-11-20T15:54:33.538+01:00][DEBUG][plugins.fleet] Deploying policies: policy-elastic-agent-on-cloud:4, test-policy-default-1:4, ...
```
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
The solution must be aware of the onboarding token from the cloud
onboarding flow. With this information, it can redirect our users to the
appropriate onboarding flow in Kibana based on their token. We need to
create an API in kibana for cloud to save some basic data.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Close https://github.com/elastic/kibana/issues/193473
Close https://github.com/elastic/kibana/issues/193474
This PR utilize the documentation packages that are build via the tool
introduced by https://github.com/elastic/kibana/pull/193847, allowing to
install them in Kibana and expose documentation retrieval as an LLM task
that AI assistants (or other consumers) can call.
Users can now decide to install the Elastic documentation from the
assistant's config screen, which will expose a new tool for the
assistant, `retrieve_documentation` (only implemented for the o11y
assistant in the current PR, shall be done for security as a follow up).
For more information, please refer to the self-review.
## General architecture
<img width="1118" alt="Screenshot 2024-10-17 at 09 22 32"
src="https://github.com/user-attachments/assets/3df8c30a-9ccc-49ab-92ce-c204b96d6fc4">
## What this PR does
Adds two plugin:
- `productDocBase`: contains all the logic related to product
documentation installation, status, and search. This is meant to be a
"low level" components only responsible for this specific part.
- `llmTasks`: an higher level plugin that will contain various LLM tasks
to be used by assistants and genAI consumers. The intent is not to have
a single place to put all llm tasks, but more to have a default place
where we can introduce new tasks from. (fwiw, the `nlToEsql` task will
probably be moved to that plugin).
- Add a `retrieve_documentation` tool registration for the o11y
assistant
- Add a component on the o11y assistant configuration page to install
the product doc
(wiring the feature to the o11y assistant was done for testing purposes
mostly, any addition / changes / enhancement should be done by the
owning team - either in this PR or as a follow-up)
## What is NOT included in this PR:
- Wire product base feature to the security assistant (should be done by
the owning team as a follow-up)
- installation
- utilization as tool
- FTR tests: this is somewhat blocked by the same things we need to
figure out for https://github.com/elastic/kibana-team/issues/1271
## Screenshots
### Installation from o11y assistant configuration page
<img width="1476" alt="Screenshot 2024-10-17 at 09 41 24"
src="https://github.com/user-attachments/assets/31daa585-9fb2-400a-a2d1-5917a262367a">
### Example of output
#### Without product documentation installed
<img width="739" alt="Screenshot 2024-10-10 at 09 59 41"
src="https://github.com/user-attachments/assets/993fb216-6c9a-433f-bf44-f6e383d20d9d">
#### With product documentation installed
<img width="718" alt="Screenshot 2024-10-10 at 09 55 38"
src="https://github.com/user-attachments/assets/805ea4ca-8bc9-4355-a434-0ba81f8228a9">
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Alex Szabo <alex.szabo@elastic.co>
Co-authored-by: Matthias Wilhelm <matthias.wilhelm@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
close https://github.com/elastic/kibana/issues/194165
close https://github.com/elastic/kibana-team/issues/1245
### User-facing
<img width="1680" alt="image"
src="https://github.com/user-attachments/assets/6df4ee9f-1b4d-404c-a764-592998a1d430">
This PRs adds a new tab in the editor history component. You can star
your query from the history and then you will see it in the Starred
list. The started queries are scoped to a user and a space.
### Server
To allow starring ESQL query, this PR extends [favorites
service](https://github.com/elastic/kibana/pull/189285) with ability to
store metadata in addition to an id. To make metadata strict and in
future to support proper metadata migrations if needed, metadata needs
to be defined as schema:
```
plugins.contentManagement.favorites.registerFavoriteType('esql_query', {
typeMetadataSchema: schema.object({ query: schema.string(), timeRange:...., etc... }),
})
```
Notable changes:
- Add support for registering a favorite type and a schema for favorite
type metadata. Previosly the `dashboard` type was the only supported
type and was hardcoded
- Add `favoriteMetadata` property to a saved object mapping and make it
`enabled:false` we don't want to index it, but just want to store
metadata in addition to an id.
[code](https://github.com/elastic/kibana/pull/198362/files#diff-d1a39e36f1de11a1110520d7607e6aee7d506c76626993842cb58db012b760a2R74-R87)
- Add a 100 favorite items limit (per type per space per user). Just do
it for sanity to prevent too large objects due to metadata stored in
addtion to ids.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
Co-authored-by: Stratoula Kalafateli <stratoula1@gmail.com>
## Summary
Fixes https://github.com/elastic/kibana/issues/190801 !!
Handle private locations simultaneous edits !!
Registered a new saved object to handle private locations properly.
Instead of using a singleton, now each private location will be
represented by it's own saved object.
### Existing private locations
When we are doing any write operation, we migrate them to new kind of
saved object and remove the legacy saved object type.
### Testing
- Create multiple private locations on main
- Switch to branch and create few more locations
- Make sure all private locations are editable, deleteable and have been
migrated to new saved object types in the course
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/189506
Testing steps:
- enable deleting unenrolled agents by adding
`xpack.fleet.enableDeleteUnenrolledAgents: true` to `kibana.dev.yml` or
turn it on on the UI
- add some unenroll agents with the helper script
```
cd x-pack/plugins/fleet
node scripts/create_agents/index.js --status unenrolled --count 10
info Creating 10 agents with statuses:
info unenrolled: 10
info Batch complete, created 10 agent docs, took 0, errors: false
info All batches complete. Created 10 agents in total. Goodbye!
```
- restart kibana or wait for the task to run and verify that the
unenrolled agents were deleted
```
[2024-10-08T16:14:45.152+02:00][DEBUG][plugins.fleet.fleet:delete-unenrolled-agents-task:0.0.5] [DeleteUnenrolledAgentsTask] Executed deletion of 10 unenrolled agents
[2024-10-08T16:14:45.153+02:00][INFO ][plugins.fleet.fleet:delete-unenrolled-agents-task:0.0.5] [DeleteUnenrolledAgentsTask] runTask ended: success
```
Added to UI settings:
<img width="1057" alt="image"
src="https://github.com/user-attachments/assets/2c9279f9-86a8-4630-a6cd-5aaa42e05fe7">
If the flag is preconfigured, disabled update on the UI with a tooltip:
<img width="1009" alt="image"
src="https://github.com/user-attachments/assets/45041020-6447-4295-995e-6848f0238f88">
The update is also prevented from the API:
<img width="2522" alt="image"
src="https://github.com/user-attachments/assets/cfbc8e21-e062-4e7f-9d08-9767fa387752">
Once the preconfiguration is removed, the UI update is allowed again.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Add the "Ouroboros" part of the entity engine:
- an enrich policy is created for each engine
- the enrich policy is executed every 30s by a kibana task, this will be
1h once we move to a 24h lookback
- create an ingest pipeline for the latest which performs the specified
field retention operations (for more detail see below)
<img width="2112" alt="Screenshot 2024-10-02 at 13 42 11"
src="https://github.com/user-attachments/assets/f727607f-2e0a-4056-a51e-393fb2a97a95">
<details>
<summary> Expand for example host entity </summary>
```
{
"@timestamp": "2024-10-01T12:10:46.000Z",
"host": {
"name": "host9",
"hostname": [
"host9"
],
"domain": [
"test.com"
],
"ip": [
"1.1.1.1",
"1.1.1.2",
"1.1.1.3"
],
"risk": {
"calculated_score": "70.0",
"calculated_score_norm": "27.00200653076172",
"calculated_level": "Low"
},
"id": [
"1234567890abcdef"
],
"type": [
"server"
],
"mac": [
"AA:AA:AA:AA:AA:AB",
"aa:aa:aa:aa:aa:aa",
"AA:AA:AA:AA:AA:AC"
],
"architecture": [
"x86_64"
]
},
"asset": {
"criticality": "low_impact"
},
"entity": {
"name": "host9",
"id": "kP/jiFHWSwWlO7W0+fGWrg==",
"source": [
"risk-score.risk-score-latest-default",
".asset-criticality.asset-criticality-default",
".ds-logs-testlogs1-default-2024.10.01-000001",
".ds-logs-testlogs2-default-2024.10.01-000001",
".ds-logs-testlogs3-default-2024.10.01-000001"
],
"type": "host"
}
}
```
</details>
### Field retention operators
First some terminology:
- **latest value** - the value produced by the transform which
represents the latest vioew of a given field in the transform lookback
period
- **enrich value** - the value added to the document by the enrich
policy, this represents the last value of a field outiside of the
transform lookback window
We hope that this will one day be merged into the entity manager
framework so I've tried to abstract this as much as possible. A field
retention operator specifies how we should choose a value for a field
when looking at the latest value and the enrich value.
### Collect values
Collect unique values in an array, first taking from the latest values
and then filling with enrich values up to maxLength.
```
{
operation: 'collect_values',
field: 'host.ip',
maxLength: 10
}
```
### Prefer newest value
Choose the latest value if present, otherwise choose the enrich value.
```
{
operation: 'prefer_newest_value',
field: 'asset.criticality'
}
```
### Prefer oldest value
Choose the enrich value if it is present, otherwise choose latest.
```
{
operation: 'prefer_oldest_value',
field: 'first_seen_timestamp'
}
```
## Test instructions
We currently require extra permissions for the kibana system user for
this to work, so we must
### 1. Get Elasticsearch running from source
This prototype requires a custom branch of elasticsearch in order to
give the kibana system user more privileges.
#### Step 1 - Clone the prototype branch
The elasticsearch branch is at
https://github.com/elastic/elasticsearch/tree/entity-store-permissions.
Or you can use [github command line](https://cli.github.com/) to
checkout my draft PR:
```
gh pr checkout 113942
```
#### Step 2 - Install Java
Install [homebrew](https://brew.sh/) if you do not have it.
```
brew install openjdk@21
sudo ln -sfn /opt/homebrew/opt/openjdk@21/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk-21.jdk
```
#### Step 3 - Run elasticsearch
This makes sure your data stays between runs of elasticsearch, and that
you have platinum license features
```
./gradlew run --data-dir /tmp/elasticsearch-repo --preserve-data -Drun.license_type=trial
```
### 2. Get Kibana Running
#### Step 1 - Connect kibana to elasticsearch
Set this in your kibana config:
```
elasticsearch.username: elastic-admin
elasticsearch.password: elastic-password
```
Now start kibana and you should have connected to the elasticsearch you
made.
### 3. Initialise entity engine and send data!
- Initialise the host or user engine (or both)
```
curl -H 'Content-Type: application/json' \
-X POST \
-H 'kbn-xsrf: true' \
-H 'elastic-api-version: 2023-10-31' \
-d '{}' \
http:///elastic:changeme@localhost:5601/api/entity_store/engines/host/init
```
- use your favourite data generation tool to create data, maybe
https://github.com/elastic/security-documents-generator
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
- The changes included in this PR allows the alerts in closed state to
be included in risk score calculation.
- It also includes the changes to backfill existing data with the
required key so that older alerts could also be considered for risk
score calculation if need be.
- Unit tests and integration tests are also included for the
changes.Tests for backfill changes are not included in this PR
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Resolves https://github.com/elastic/kibana/issues/153950.
This PR implements a UI to configure advanced Elastic Agent monitoring
options under agent policy settings. These advanced options include
enabling HTTP monitoring endpoints and various options for agent
diagnostics. They are shown under an a toggle under the existing agent
monitoring logs and metrics collection options:
<img width="1326" alt="image"
src="https://github.com/user-attachments/assets/ac8cbe00-d838-4c9a-8a35-3dbf31222dc9">
If the base HTTP monitoring endpoint is not enabled, the rest of the
HTTP options are disabled:
<img width="1328" alt="image"
src="https://github.com/user-attachments/assets/2eac787c-3055-4862-b3eb-2566a39ee86c">
The following new fields are added to agent policy schema to support
this:
```
monitoring_http
monitoring_pprof_enabled
monitoring_diagnostics
```
This work supersedes the previous `HTTP monitoring endpoint` options
under `Advanced Settings` at the bottom of the page. Any previous
configuration under an agent policy's
`advanced_settings.agent_monitoring_http` saved object field are
migrated over to the new `monitoring_http` field and the old field is
deleted. See the migration fn `backfillAgentPolicyToV4`.
These new options are compiled to agent yaml like this:
```yml
agent:
monitoring:
enabled: true
use_output: default
logs: true
metrics: true
traces: true
namespace: default
pprof:
enabled: true
http:
enabled: true
host: localhost
port: 6791
diagnostics:
limit:
interval: 1m
burst: 1
uploader:
max_retries: 10
init_dur: 1s
max_dur: 10m
```
Summarize your PR. If it involves visual changes include a screenshot or
gif.
### To-do
- [x] API integration tests
- [x] Full manual test of SO migration
- [x] Full manual test with agent using these settings
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- https://github.com/elastic/ingest-docs/issues/1333
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This PR introduces the following API routes for setting up Entity Store
"engines":
<meta charset="utf-8"><b style="font-weight:normal;"
id="docs-internal-guid-9410c5d7-7fff-e873-6830-887939a306fb"><div
dir="ltr" style="margin-left:-0.75pt;" align="left">
Initialise Engine | POST /api/entity_store/engines/<entity_type>/init
-- | --
Start Engine | POST /api/entity_store/engines/<entity_type>/start
Stop Engine | POST /api/entity_store/engines/<entity_type>/stop
Delete Engine | DELETE /api/entity_store/engines/<entity_type>
Get engine | GET /api/entity_store/engines/<entity_type>
List Engines | GET /api/entity_store/engines
</div></b>
The PR includes the following:
- Adding the `EntityManager` plugin (see elastic/obs-entities) as a
dependency of the Security Solution
- The OpenAPI schemas for the new routes
- The actual Kibana side endpoints
- A `Saved Object` to track the installed engines
- A new `EntityStoreDataClient`
- A new feature flag `entityStoreEngineRoutesEnabled`
### How to test
1. Add some host/user data
* Easiest is to use
[elastic/security-data-generator](https://github.com/elastic/security-documents-generator)
2. Make sure to add `entityStoreEngineRoutesEnabled` under
`xpack.securitySolution.enableExperimental` in your `kibana.dev.yml`
3. In kibana dev tools or your terminal, call the `INIT` route for
either `user` or `host`.
4. You should now see 2 transforms in kibana. Make sure to re-trigger
them if needed so they process the documents.
5. Check that new entities have been observed by querying the new
entities index via:
* `GET .entities.v1.latest.ea*/_search`
6. Check the other endpoints are working (`START`, `STOP`, `LIST`, etc)
7. Calling `DELETE` should remove the transforms
Implements https://github.com/elastic/security-team/issues/10230
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Add telemetry to favorites feature
https://github.com/elastic/kibana/pull/189285
- Adds UI usage counter telemetry, increase the counter when favorite /
unfavorite is clicked
- Add snapshot telemetry:
- total "favorite" object in the deployment
- total users+spaces count combination who have used the favorites
feature
- avg per user per space (only counts those users who favorited at least
once)
- max favorites objects per user per space
Unfortunately, for snapshot telemetry, I had to add fields to kibana
mapping. We didn't need them for a feature, but I didn't realize that
will have to add them to a mapping. Not sure if there is a better way
## Summary
Resolves#143905. This PR adds support for integration-level outputs.
This means that different integrations within the same agent policy can
now be configured to send data to different locations. This feature is
gated behind `enterprise` level subscription.
For each input, the agent policy will configure sending data to the
following outputs in decreasing order of priority:
1. Output set specifically on the integration policy
2. Output set specifically on the integration's parent agent policy
(including the case where an integration policy belongs to multiple
agent policies)
3. Global default data output set via Fleet Settings
Integration-level outputs will respect the same rules as agent
policy-level outputs:
- Certain integrations are disallowed from using certain output types,
attempting to add them to each other via creation, updating, or
"defaulting", will fail
- `fleet-server`, `synthetics`, and `apm` can only use same-cluster
Elasticsearch output
- When an output is deleted, any integrations that were specifically
using it will "clear" their output configuration and revert back to
either `#2` or `#3` in the above list
- When an output is edited, all agent policies across all spaces that
use it will be bumped to a new revision, this includes:
- Agent policies that have that output specifically set in their
settings (existing behavior)
- Agent policies that contain integrations which specifically has that
output set (new behavior)
- When a proxy is edited, the same new revision bump above will apply
for any outputs using that proxy
The final agent policy YAML that is generated will have:
- `outputs` block that includes:
- Data and monitoring outputs set at the agent policy level (existing
behavior)
- Any additional outputs set at the integration level, if they differ
from the above
- `outputs_permissions` block that includes permissions for each
Elasticsearch output depending on which integrations and/or agent
monitoring are assigned to it
Integration policies table now includes `Output` column. If the output
is defaulting to agent policy-level output, or global setting output, a
tooltip is shown:
<img width="1392" alt="image"
src="https://github.com/user-attachments/assets/5534716b-49b5-402a-aa4a-4ba6533e0ca8">
Configuring an integration-level output is done under Advanced options
in the policy editor. Setting to the blank value will "clear" the output
configuration. The list of available outputs is filtered by what outputs
are available for that integration (see above):
<img width="799" alt="image"
src="https://github.com/user-attachments/assets/617af6f4-e8f8-40b1-b476-848f8ac96e76">
An example of failure: ES output cannot be changed to Kafka while there
is an integration
<img width="1289" alt="image"
src="https://github.com/user-attachments/assets/11847eb5-fd5d-4271-8464-983d7ab39218">
## TODO
- [x] Adjust side effects of editing/deleting output when policies use
it across different spaces
- [x] Add API integration tests
- [x] Update OpenAPI spec
- [x] Create doc issue
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
