# Summary
Adds a new API deprecations feature inside core.
This feature enabled plugin developers to mark their versioned and
unversioned public routes as deprecated.
These deprecations will be surfaced to the users through UA to help them
understand the deprecation and address it before upgrading. This PR also
surfaces these deprecations to UA.
Closes https://github.com/elastic/kibana/issues/117241
1. Core service to flag deprecated routes
2. UA code to surface and resolve deprecated routes
## Flagging a deprecated Route
### The route deprecation option
We have three types of route deprecations:
- `type: bump`: A version bump deprecation means the API has a new
version and the current version will be removed in the future in favor
of the newer version.
- `type: remove`: This API will be completely removed. You will no
longer be able to use it in the future.
- `type: migrate`: This API will be migrated to a different API and will
be removed in the future in favor of the other API.
All route deprecations expect a documentation link to help users
navigate. We might add a generic documentation link and drop this
requirement in the future but for now this is required.
### Deprecated Route Example
Full examples can be found in the `routing_example` example plugin
located in this directory:
`examples/routing_example/server/routes/deprecated_routes`
```ts
router[versioned?].get(
{
path: '/',
options: {
deprecated: {
documentationUrl: 'https://google.com',
severity: 'warning',
reason: {
type: 'bump',
newApiVersion: '2024-10-13',
},
},
},
},
async (context, req, res) => {
...
```
## Surfaced API deprecations in UA
The list of deprecated APIs will be listed inside Kibana deprecations
along with the already supported config deprecations.
<img width="1728" alt="image"
src="https://github.com/user-attachments/assets/5bece704-b80b-4397-8ba2-6235f8995e4a">
Users can click on the list item to learn more about each deprecation
and mark it as resolved
<img width="1476" alt="image"
src="https://github.com/user-attachments/assets/91c9207b-b246-482d-a5e4-21d0c61582a8">
### Marking as resolved
Users can click on mark as resolved button in the UA to hide the
deprecation from the Kiban deprecations list.
We keep track on when this button was clicked and how many times the API
has been called. If the API is called again the deprecation will
re-appear inside the list. We might add a feature in the future to
permenantly supress the API deprecation from showing in the list through
a configuration (https://github.com/elastic/kibana/issues/196089)
If the API has been marked as resolved before we show this in the flyout
message:
> The API GET /api/deprecations/ has been called 25 times. The last time
the API was called was on Monday, October 14, 2024 1:08 PM +03:00.
> The api has been called 2 times since the last time it was marked as
resolved on Monday, October 14, 2024 1:08 PM +03:00
Once marked as resolved the flyout exists and we show this to the user
until they refresh the page
<img width="1453" alt="image"
src="https://github.com/user-attachments/assets/8bb5bc8b-d1a3-478f-9489-23cfa7db6350">
## Telemetry:
We keep track of 2 new things for telemetry purposes:
1. The number of times the deprecated API has been called
2. The number of times the deprecated API has been resolved (how many
times the mark as resolved button in UA was clicked)
## Code review
- [x] Core team is expected to review the whole PR
- [ ] Docs team to review the copy and update the UA displayed texts
(title, description, and manual steps)
- [x] kibana-management team is expected to review the UA code changes
and UI
- [ ] A few teams are only required to approve this PR and update their
`deprecated: true` route param to the new deprecationInfo object we now
expect. There is an issue tracker to address those in separate PRs later
on: https://github.com/elastic/kibana/issues/196095
## Testing
Run kibana locally with the test example plugin that has deprecated
routes
```
yarn start --plugin-path=examples/routing_example --plugin-path=examples/developer_examples
```
The following comprehensive deprecated routes examples are registered
inside the folder:
`examples/routing_example/server/routes/deprecated_routes`
Run them in the console to trigger the deprecation condition so they
show up in the UA:
```
# Versioned routes: Version 1 is deprecated
GET kbn:/api/routing_example/d/versioned?apiVersion=1
GET kbn:/api/routing_example/d/versioned?apiVersion=2
# Non-versioned routes
GET kbn:/api/routing_example/d/removed_route
POST kbn:/api/routing_example/d/migrated_route
{}
```
1. You can also mark as deprecated in the UA to remove the deprecation
from the list.
2. Check the telemetry response to see the reported data about the
deprecated route.
3. Calling version 2 of the API does not do anything since it is not
deprecated unlike version `1` (`GET
kbn:/api/routing_example/d/versioned?apiVersion=2`)
4. Internally you can see the deprecations counters from the dev console
by running the following:
```
GET .kibana_usage_counters/_search
{
"query": {
"bool": {
"should": [
{"match": { "usage-counter.counterType": "deprecated_api_call:total"}},
{"match": { "usage-counter.counterType": "deprecated_api_call:resolved"}},
{"match": { "usage-counter.counterType": "deprecated_api_call:marked_as_resolved"}}
]
}
}
}
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: florent-leborgne <florent.leborgne@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Closes#194933Closes#192282
## Summary
This PR updates the serverless project yml files to
- enable custom roles for Elasticsearch and Security projects
- enable multiple spaces (max 100) for all serverless project types
### Tests
Additionally, this PR adjust the serverless test suites. Originally,
testing of roles and spaces endpoints was achieved from the feature flag
test config. Now that these features are enabled by default, the tests
have been migrated to the standard serverless test configs.
Affected tests:
-
x-pack/test_serverless/api_integration/test_suites/common/management/spaces.ts
-
x-pack/test_serverless/api_integration/test_suites/common/platform_security/authorization.ts
-
x-pack/test_serverless/functional/test_suites/common/platform_security/navigation/management_nav_cards.ts
-
x-pack/test_serverless/functional/test_suites/common/platform_security/roles.ts
-
x-pack/test_serverless/functional/test_suites/common/spaces/spaces_management.ts
-
x-pack/test_serverless/functional/test_suites/common/spaces/spaces_selection.ts
- Feature flag configs/indices
- Project specific configs/indices
- Base serverless config
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
### Entity store enablement UI
This PR adds a UI to enable the Entity Store.
### How to test
1. Enable `entityStoreEnabled` experimental feature flag
2. Navigate to `Security > Dashboards > Entity Analytics`
3. Work through the distinct flows to enable the store
* For example, choose to enable risk score together with the store
4. Navigate to `Security > Manage > Entity Store` to start/stop the
store
5. Validate that the appropriate transforms and pipelines have been
initialized and have the correct status (for example, via the Stack
Management UI)
EDIT:
Enablement flow screenshots:
#### Enable both risk score and entity store
#### Enable Risk score only (Entity store already enabled)
#### Modal to choose what to enable
#### New Entity Store management page
---------
Co-authored-by: jaredburgettelastic <jared.burgett@elastic.co>
Co-authored-by: machadoum <pablo.nevesmachado@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Mark Hopkin <mark.hopkin@elastic.co>
Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com>
## Summary
* Update the Entity Store to retrieve indices from the security solution
data view.
* Create a new API that updates all installed entity engine indices
(`api/entity_store/engines/apply_dataview_indices`)
### How to test it?
* Install the entity store
* Check if the transform index has the security solutions data view
indices
* Call `apply_dataview_indices` API; it should not return changes
* Update the security solution data view indices
* Call `apply_dataview_indices` API and if the API response contains the
updated indices
* Check if the transform index also got updated
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This change is the implementation of the `Kibana Privilege Migrations`
proposal/RFC and provides a framework that allows developers to replace
an existing feature with a new one that has the desired configuration
while teaching the platform how the privileges of the deprecated feature
can be represented by non-deprecated ones. This approach avoids
introducing breaking changes for users who still rely on the deprecated
privileges in their existing roles and any automation.
Among the use cases the framework is supposed to handle, the most common
are the following:
* Changing a feature ID from `Alpha` to `Beta`
* Splitting a feature `Alpha` into two features, `Beta` and `Gamma`
* Moving a capability between privileges within a feature (top-level or
sub-feature)
* Consolidating capabilities across independent features
## Scope
This PR includes only the core functionality proposed in the RFC and
most of the necessary guardrails (tests, early validations, etc.) to
help engineers start planning and implementing their migrations as soon
as possible. The following functionality will be added in follow-ups or
once we collect enough feedback:
* Telemetry
* Developer documentation
* UI enhancements (highlighting roles with deprecated privileges and
manual migration actions)
## Framework
The steps below use a scenario where a feature `Alpha` should be split
into two other features `Beta` and `Gamma` as an example.
### Step 1: Create new features with the desired privileges
First of all, define new feature or features with the desired
configuration as you'd do before. There are no constraints here.
<details>
<summary>Click to see the code</summary>
```ts
deps.features.registerKibanaFeature({
id: 'feature_beta',
name: 'Feature Beta',
privileges: {
all: {
savedObject: { all: ['saved_object_1'], read: [] },
ui: ['ui_all'],
api: ['api_all'],
… omitted for brevity …
},
read: {
savedObject: { all: [], read: ['saved_object_1'] },
ui: ['ui_read'],
api: ['api_read'],
… omitted for brevity …
},
},
… omitted for brevity …
});
deps.features.registerKibanaFeature({
id: 'feature_gamma',
name: 'Feature Gamma',
privileges: {
all: {
savedObject: { all: ['saved_object_2'], read: [] },
ui: ['ui_all'],
// Note that Feature Gamma, unlike Features Alpha and Beta doesn't provide any API access tags
… omitted for brevity …
},
read: {
savedObject: { all: [], read: ['saved_object_2'] },
ui: ['ui_read'],
// Note that Feature Gamma, unlike Features Alpha and Beta doesn't provide any API access tags
… omitted for brevity …
},
},
… omitted for brevity …
});
```
</details>
### Step 2: Mark existing feature as deprecated
Once a feature is marked as deprecated, it should essentially be treated
as frozen for backward compatibility reasons. Deprecated features will
no longer be available through the Kibana role management UI and will be
replaced with non-deprecated privileges.
Deprecated privileges will still be accepted if the role is created or
updated via the Kibana role management APIs to avoid disrupting existing
user automation.
To avoid breaking existing roles that reference privileges provided by
the deprecated features, Kibana will continue registering these
privileges as Elasticsearch application privileges.
<details>
<summary>Click to see the code</summary>
```ts
deps.features.registerKibanaFeature({
// This is a new `KibanaFeature` property available during feature registration.
deprecated: {
// User-facing justification for privilege deprecation that we can display
// to the user when we ask them to perform role migration.
notice: i18n.translate('xpack.security...', {
defaultMessage: "Feature Alpha is deprecated, refer to {link}...",
values: { link: docLinks.links.security.deprecatedFeatureAlpha },
})
},
// Feature id should stay unchanged, and it's not possible to reuse it.
id: 'feature_alpha',
name: 'Feature Alpha (DEPRECATED)',
privileges: {
all: {
savedObject: { all: ['saved_object_1', 'saved_object_2'], read: [] },
ui: ['ui_all'],
api: ['api_all'],
… omitted for brevity …
},
read: {
savedObject: { all: [], read: ['saved_object_1', 'saved_object_2'] },
ui: ['ui_read'],
api: ['api_read'],
… omitted for brevity …
},
},
… omitted for brevity …
});
```
</details>
### Step 3: Map deprecated feature’s privileges to the privileges of the
non-deprecated features
The important requirement for a successful migration from a deprecated
feature to a new feature or features is that it should be possible to
express **any combination** of the deprecated feature and sub-feature
privileges with the feature or sub-feature privileges of non-deprecated
features. This way, while editing a role with deprecated feature
privileges in the UI, the admin will be interacting with new privileges
as if they were creating a new role from scratch, maintaining
consistency.
The relationship between the privileges of the deprecated feature and
the privileges of the features that are supposed to replace them is
expressed with a new `replacedBy` property available on the privileges
of the deprecated feature.
<details>
<summary>Click to see the code</summary>
```ts
deps.features.registerKibanaFeature({
// This is a new `KibanaFeature` property available during feature registration.
deprecated: {
// User-facing justification for privilege deprecation that we can display
// to the user when we ask them to perform role migration.
notice: i18n.translate('xpack.security...', {
defaultMessage: "Feature Alpha is deprecated, refer to {link}...",
values: { link: docLinks.links.security.deprecatedFeatureAlpha },
})
},
// Feature id should stay unchanged, and it's not possible to reuse it.
id: 'feature_alpha',
name: 'Feature Alpha (DEPRECATED)',
privileges: {
all: {
savedObject: { all: ['saved_object_1', 'saved_object_2'], read: [] },
ui: ['ui_all'],
api: ['api_all'],
replacedBy: [
{ feature: 'feature_beta', privileges: ['all'] },
{ feature: 'feature_gamma', privileges: ['all'] },
],
… omitted for brevity …
},
read: {
savedObject: { all: [], read: ['saved_object_1', 'saved_object_2'] },
ui: ['ui_read'],
api: ['api_read'],
replacedBy: [
{ feature: 'feature_beta', privileges: ['read'] },
{ feature: 'feature_gamma', privileges: ['read'] },
],
… omitted for brevity …
},
},
… omitted for brevity …
});
```
</details>
### Step 4: Adjust the code to rely only on new, non-deprecated features
Special care should be taken if the replacement privileges cannot reuse
the API access tags from the deprecated privileges and introduce new
tags that will be applied to the same API endpoints. In this case,
developers should replace the API access tags of the deprecated
privileges with the corresponding tags provided by the replacement
privileges. This is necessary because API endpoints can only be accessed
if the user privileges cover all the tags listed in the API endpoint
definition, and without these changes, existing roles referencing
deprecated privileges won’t be able to access those endpoints.
The UI capabilities are handled slightly differently because they are
always prefixed with the feature ID. When migrating to new features with
new IDs, the code that interacts with UI capabilities will be updated to
use these new feature IDs.
<details>
<summary>Click to see the code</summary>
```ts
// BEFORE deprecation/migration
// 1. Feature Alpha defition (not deprecated yet)
deps.features.registerKibanaFeature({
id: 'feature_alpha',
privileges: {
all: {
api: ['api_all'],
… omitted for brevity …
},
},
… omitted for brevity …
});
// 2. Route protected by `all` privilege of the Feature Alpha
router.post(
{ path: '/api/domain/my_api', options: { tags: ['access:api_all'] } },
async (_context, request, response) => {}
);
// AFTER deprecation/migration
// 1. Feature Alpha defition (deprecated, with updated API tags)
deps.features.registerKibanaFeature({
deprecated: …,
id: 'feature_alpha',
privileges: {
all: {
api: ['api_all_v2'],
replacedBy: [
{ feature: 'feature_beta', privileges: ['all'] },
],
… omitted for brevity …
},
},
… omitted for brevity …
});
// 2. Feature Beta defition (new)
deps.features.registerKibanaFeature({
id: 'feature_beta',
privileges: {
all: {
api: ['api_all_v2'],
… omitted for brevity …
}
},
… omitted for brevity …
});
// 3. Route protected by `all` privilege of the Feature Alpha OR Feature Beta
router.post(
{ path: '/api/domain/my_api', options: { tags: ['access:api_all_v2'] } },
async (_context, request, response) => {}
);
----
// ❌ Old client-side code (supports only deprecated privileges)
if (capabilities.feature_alpha.ui_all) {
… omitted for brevity …
}
// ✅ New client-side code (will work for **both** new and deprecated privileges)
if (capabilities.feature_beta.ui_all) {
… omitted for brevity …
}
```
</details>
## How to test
The code introduces a set of API integration tests that are designed to
validate whether the privilege mapping between deprecated and
replacement privileges maintains backward compatibility.
You can run the test server with the following config to register a
number of [example deprecated
features](https://github.com/elastic/kibana/pull/186800/files#diff-d887981d43bbe30cda039340b906b0fa7649ba80230be4de8eda326036f10f6fR20-R49)(`x-pack/test/security_api_integration/plugins/features_provider/server/index.ts`)
and the features that replace them, to see the framework in action:
```bash
node scripts/functional_tests_server.js --config x-pack/test/security_api_integration/features.config.ts
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/189506
Testing steps:
- enable deleting unenrolled agents by adding
`xpack.fleet.enableDeleteUnenrolledAgents: true` to `kibana.dev.yml` or
turn it on on the UI
- add some unenroll agents with the helper script
```
cd x-pack/plugins/fleet
node scripts/create_agents/index.js --status unenrolled --count 10
info Creating 10 agents with statuses:
info unenrolled: 10
info Batch complete, created 10 agent docs, took 0, errors: false
info All batches complete. Created 10 agents in total. Goodbye!
```
- restart kibana or wait for the task to run and verify that the
unenrolled agents were deleted
```
[2024-10-08T16:14:45.152+02:00][DEBUG][plugins.fleet.fleet:delete-unenrolled-agents-task:0.0.5] [DeleteUnenrolledAgentsTask] Executed deletion of 10 unenrolled agents
[2024-10-08T16:14:45.153+02:00][INFO ][plugins.fleet.fleet:delete-unenrolled-agents-task:0.0.5] [DeleteUnenrolledAgentsTask] runTask ended: success
```
Added to UI settings:
<img width="1057" alt="image"
src="https://github.com/user-attachments/assets/2c9279f9-86a8-4630-a6cd-5aaa42e05fe7">
If the flag is preconfigured, disabled update on the UI with a tooltip:
<img width="1009" alt="image"
src="https://github.com/user-attachments/assets/45041020-6447-4295-995e-6848f0238f88">
The update is also prevented from the API:
<img width="2522" alt="image"
src="https://github.com/user-attachments/assets/cfbc8e21-e062-4e7f-9d08-9767fa387752">
Once the preconfiguration is removed, the UI update is allowed again.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Resolves https://github.com/elastic/kibana/issues/188043
This PR adds new connector which is define integration with Elastic
Inference Endpoint via [Inference
APIs](https://www.elastic.co/guide/en/elasticsearch/reference/current/inference-apis.html)
The lifecycle of the Inference Endpoint are managed by the connector
registered handlers:
- `preSaveHook` -
[create](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-inference-api.html)
new Inference Endpoint in the connector create mode (`isEdit === false`)
and
[delete](https://www.elastic.co/guide/en/elasticsearch/reference/current/delete-inference-api.html)+[create](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-inference-api.html)
in the connector edit mode (`isEdit === true`)
- `postSaveHook` - check if the connector SO was created/updated and if
not removes Inference Endpoint from preSaveHook
- `postDeleteHook` -
[delete](https://www.elastic.co/guide/en/elasticsearch/reference/current/delete-inference-api.html)
Inference Endpoint if connector was deleted.
In the Kibana Stack Management Connectors, its represented with the new
card (Technical preview badge):
<img width="1261" alt="Screenshot 2024-09-27 at 2 11 12 PM"
src="https://github.com/user-attachments/assets/dcbcce1f-06e7-4d08-8b77-0ba4105354f8">
To simplify the future integration with AI Assistants, the Connector
consists from the two main UI parts: provider selector and required
provider settings, which will be always displayed
<img width="862" alt="Screenshot 2024-10-07 at 7 59 09 AM"
src="https://github.com/user-attachments/assets/87bae493-c642-479e-b28f-6150354608dd">
and Additional options, which contains optional provider settings and
Task Type configuration:
<img width="861" alt="Screenshot 2024-10-07 at 8 00 15 AM"
src="https://github.com/user-attachments/assets/2341c034-6198-4731-8ce7-e22e6c6fb20f">
subActions corresponds to the different taskTypes Inference API
supports. Each of the task type has its own Inference Perform params.
Currently added:
- completion & completionStream
- rerank
- text_embedding
- sparse_embedding
Follow up work:
1. Collapse/expand Additional options, when the connector flyout/modal
has AI Assistant as a context (path through the extending context
implementation on the connector framework level)
2. Add support for additional params for Completion subAction to be able
to path functions
3. Add support for tokens usage Dashboard, when inference API will
include the used tokens count in the response
4. Add functionality and UX for migration from existing specific AI
connectors to the Inference connector with proper provider and
completion task
5. Integrate Connector with the AI Assistants
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
Co-authored-by: Steph Milovic <stephanie.milovic@elastic.co>
## Summary
Add the "Ouroboros" part of the entity engine:
- an enrich policy is created for each engine
- the enrich policy is executed every 30s by a kibana task, this will be
1h once we move to a 24h lookback
- create an ingest pipeline for the latest which performs the specified
field retention operations (for more detail see below)
<img width="2112" alt="Screenshot 2024-10-02 at 13 42 11"
src="https://github.com/user-attachments/assets/f727607f-2e0a-4056-a51e-393fb2a97a95">
<details>
<summary> Expand for example host entity </summary>
```
{
"@timestamp": "2024-10-01T12:10:46.000Z",
"host": {
"name": "host9",
"hostname": [
"host9"
],
"domain": [
"test.com"
],
"ip": [
"1.1.1.1",
"1.1.1.2",
"1.1.1.3"
],
"risk": {
"calculated_score": "70.0",
"calculated_score_norm": "27.00200653076172",
"calculated_level": "Low"
},
"id": [
"1234567890abcdef"
],
"type": [
"server"
],
"mac": [
"AA:AA:AA:AA:AA:AB",
"aa:aa:aa:aa:aa:aa",
"AA:AA:AA:AA:AA:AC"
],
"architecture": [
"x86_64"
]
},
"asset": {
"criticality": "low_impact"
},
"entity": {
"name": "host9",
"id": "kP/jiFHWSwWlO7W0+fGWrg==",
"source": [
"risk-score.risk-score-latest-default",
".asset-criticality.asset-criticality-default",
".ds-logs-testlogs1-default-2024.10.01-000001",
".ds-logs-testlogs2-default-2024.10.01-000001",
".ds-logs-testlogs3-default-2024.10.01-000001"
],
"type": "host"
}
}
```
</details>
### Field retention operators
First some terminology:
- **latest value** - the value produced by the transform which
represents the latest vioew of a given field in the transform lookback
period
- **enrich value** - the value added to the document by the enrich
policy, this represents the last value of a field outiside of the
transform lookback window
We hope that this will one day be merged into the entity manager
framework so I've tried to abstract this as much as possible. A field
retention operator specifies how we should choose a value for a field
when looking at the latest value and the enrich value.
### Collect values
Collect unique values in an array, first taking from the latest values
and then filling with enrich values up to maxLength.
```
{
operation: 'collect_values',
field: 'host.ip',
maxLength: 10
}
```
### Prefer newest value
Choose the latest value if present, otherwise choose the enrich value.
```
{
operation: 'prefer_newest_value',
field: 'asset.criticality'
}
```
### Prefer oldest value
Choose the enrich value if it is present, otherwise choose latest.
```
{
operation: 'prefer_oldest_value',
field: 'first_seen_timestamp'
}
```
## Test instructions
We currently require extra permissions for the kibana system user for
this to work, so we must
### 1. Get Elasticsearch running from source
This prototype requires a custom branch of elasticsearch in order to
give the kibana system user more privileges.
#### Step 1 - Clone the prototype branch
The elasticsearch branch is at
https://github.com/elastic/elasticsearch/tree/entity-store-permissions.
Or you can use [github command line](https://cli.github.com/) to
checkout my draft PR:
```
gh pr checkout 113942
```
#### Step 2 - Install Java
Install [homebrew](https://brew.sh/) if you do not have it.
```
brew install openjdk@21
sudo ln -sfn /opt/homebrew/opt/openjdk@21/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk-21.jdk
```
#### Step 3 - Run elasticsearch
This makes sure your data stays between runs of elasticsearch, and that
you have platinum license features
```
./gradlew run --data-dir /tmp/elasticsearch-repo --preserve-data -Drun.license_type=trial
```
### 2. Get Kibana Running
#### Step 1 - Connect kibana to elasticsearch
Set this in your kibana config:
```
elasticsearch.username: elastic-admin
elasticsearch.password: elastic-password
```
Now start kibana and you should have connected to the elasticsearch you
made.
### 3. Initialise entity engine and send data!
- Initialise the host or user engine (or both)
```
curl -H 'Content-Type: application/json' \
-X POST \
-H 'kbn-xsrf: true' \
-H 'elastic-api-version: 2023-10-31' \
-d '{}' \
http:///elastic:changeme@localhost:5601/api/entity_store/engines/host/init
```
- use your favourite data generation tool to create data, maybe
https://github.com/elastic/security-documents-generator
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Related to #192484. This PR adding [new content package types and
schemas](https://github.com/elastic/package-spec/pull/777) so that
content packages can be returned correctly from EPR to unblock
development of those packages.
The only current content package is `kubernetes_otel`. You will need to
bump up the max allowed spec version and search with beta (prerelease)
packages enabled to find it:
```
xpack.fleet.internal.registry.spec.max: '3.4'
```
Tests will come with the rest of work for #192484
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
New internal GET `/api/endpoint/metadata/transforms` route.
Current public GET `/api/endpoint/metadata/transforms` route is set to
deprecated.
All usages across the project have been updated to consume the new
internal route.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Issue: https://github.com/elastic/kibana/issues/190018
Implement rule specific flapping support for create and update Rule API.
The new property on the rule is named `flapping`;
```
flapping: {
look_back_window: number;
status_change_threshold: number;
}
```
Also make changes in the task runner to use the rule's flapping settings
if it exists. Otherwise use the global flapping setting.
# To test
1. Go to
`x-pack/plugins/triggers_actions_ui/public/common/constants/index.ts`
and turn `IS_RULE_SPECIFIC_FLAPPING_ENABLED` to `true`
2. Create a rule with a rule specific flapping setting, generate the
alert and let it flap
3. Assert that the flapping is now using the rule specific flapping
4. Turn space flapping off
5. Assert that it no longer flaps despite having a rule specific
flapping
6. Try deleting/adding back the rule specific flapping via the UI and
verify everything works.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
New internal GET `/internal/api/endpoint/suggestions/{suggestion_type}`
route.
Current public GET `/api/endpoint/suggestions/{suggestion_type}` route
is set to deprecated.
UI uses now the internal GET
`/internal/api/endpoint/suggestions/{suggestion_type}` api route
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/184685
**Release notes**: These schema changes shouldn't be breaking, but there
were some incorrect/missing response schemas in the old openapi spec.
For example the API `POST /api/fleet/agents/{agentId}/actions` response
was incorrectly documented:
https://petstore.swagger.io/?url=https://raw.githubusercontent.com/elastic/kibana/main/x-pack/plugins/fleet/common/openapi/bundled.json#/Elastic%20Agent%20actions/new-agent-action
```
{
"body": [
0
],
"statusCode": 0,
"headers": "string"
}
```
Fixed here:
31f8cfd6ef/oas_docs/bundle.json#/Elastic%20Agent%20actions/%252Fapi%252Ffleet%252Fagents%252F%257BagentId%257D%252Factions%230
```
{
"item": {
"ack_data": "string",
"agents": [
"string"
],
"created_at": "string",
"data": "string",
"expiration": "string",
"id": "string",
"minimum_execution_duration": 0,
"namespaces": [
"string"
],
"rollout_duration_seconds": 0,
"sent_at": "string",
"source_uri": "string",
"start_time": "string",
"total": 0,
"type": "string"
}
}
```
The new spec should match the implementation accurately, and responses
are being verified when returned. Tests were added to make sure the
response schemas are correct.
If there are any bugs in the current schema, it will result in a HTTP
500 error with an error message on where the schema validation failed.
Example of an error where a field is missing:
```
{
"statusCode": 500,
"error": "Internal Server Error",
"message": "Failed output validation: [request body.items.0.name]: definition for this key is missing"
}
```
Example of an error where a field is mandatory in the schema, but not
provided in the response (missing `schema.maybe`)
```
{
"statusCode": 500,
"error": "Internal Server Error",
"message": "Failed output validation: [request body.items.0.internal]: expected value of type [boolean] but got [undefined]"
}
```
There are a few places where the validation allows unknown fields. Used
it where some fields were not included in TS types or fields are more
dynamic, e.g. fields coming from packages or elasticsearch settings.
https://github.com/search?q=repo%3Aelastic%2Fkibana+extendsDeep+path%3A%2F%5Ex-pack%5C%2Fplugins%5C%2Ffleet%5C%2Fserver%5C%2Ftypes%5C%2F%2F&type=code
```
.extendsDeep({
unknowns: 'allow',
})
```
Changes in this pr:
Remove using old `bundled.yaml` to generate oas, fixed tags.
Removed old openapi files, updated readme.
Here is the new bundle in Swagger UI:
[stateful](31f8cfd6ef/oas_docs/bundle.json)
[serverless](da72ee0093/oas_docs/bundle.serverless.json)
Updated serverless scripts too.
Updated Fleet readme:
da72ee0093/x-pack/plugins/fleet/common/openapi/README.md
Generated the new bundle by running this script locally:
```
node scripts/capture_oas_snapshot --include-path /api/fleet --update
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
*Epic:** https://github.com/elastic/security-team/issues/9401 (internal)
## Summary
This PR includes Security Solution OpenAPI domain bundles into the production OpenAPI Kibana bundle. The result Kibana bundler is expected to be published to Bump.sh manually by @lcawl.
Closes#192153
## Summary
This PR sets the spaces and roles CRUD operation HTTP API endpoints to
public in both stateful and serverless offerings, and additionally,
switches to the versioned router to register these endpoints.
Prior to this PR, the access level was not explicitly set, thus any
endpoints registered in serverless were by default internal. CRUD
operations for spaces and roles are being set to public to support the
rollout of custom roles in serverless, which coincides with enabling
multiple spaces.
### Note
- Currently, roles APIs are only available in serverless via a feature
flag (`xpack.security.roleManagementEnabled`)
- Spaces APIs are already registered in serverless, however, the maximum
number of spaces is by default 1, rendering create and delete operations
unusable. By overriding `xpack.spaces.maxSpaces` to a number greater
than 1 (stateful default is 1000), it will effectively enable use of the
spaces CRUD operations in serverless.
## Tests
-
x-pack/test_serverless/api_integration/test_suites/common/management/multiple_spaces_enabled.ts
-
x-pack/test_serverless/api_integration/test_suites/common/management/spaces.ts
-
x-pack/test_serverless/api_integration/test_suites/common/platform_security/authorization.ts
-
x-pack/test_serverless/api_integration/test_suites/common/platform_security/roles_routes_feature_flag.ts
- Unit tests for each endpoint (to account for versioned router)
- Flaky Test Runner:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7002
## Manual Testing
1. Start ES & Kibana in serverless mode with config options to enable
role management and multiple spaces
Elasticsearch:
```
xpack.security.authc.native_roles.enabled: true
```
KIbana:
```
xpack.security.roleManagementEnabled: true
xpack.spaces.maxSpaces: 100
```
3. Issue each CRUD HTTP API without including the internal origin header
('x-elastic-internal-origin') and verify you do not receive a 400 with
the message "method [get|post|put|delete] exists but is not available
with the current configuration"
4. Repeat steps 1 & 2 from the current head of main and verify that you
DO receive a 400 with the message "method [get|post|put|delete] exists
but is not available with the current configuration"
Regression testing - ensure that interfaces which leverage spaces and
roles APIs are functioning properly
- Spaces management
- Space navigation
- Roles management
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
fixes: https://github.com/elastic/security-team/issues/10235
fixes: https://github.com/elastic/security-team/issues/10237
This is the final PR for migrating over all timeline-related schemas and
types to the new generated zod schemas from our OpenAPI specs. (see
https://github.com/elastic/security-team/issues/10110)
On top of moving to the new schemas/types, this PR also cleans up usage
of now outdated types.
I'm aware of the size of this PR but rest assured, the changes are easy
to review and for most teams, only a handful of files need to be
reviewed:
```markdown
### elastic/security-defend-workflows
* x-pack/test/security_solution_endpoint/apps/endpoint/endpoint_solution_integrations.ts
### elastic/security-detection-rule-management
* x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/get_prebuilt_rules_and_timelines_status/get_prebuilt_rules_and_timelines_status_route.ts
* x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/perform_timelines_installation.ts
### elastic/security-detections-response
* x-pack/test/security_solution_cypress/cypress/objects/timeline.ts
### elastic/security-engineering-productivity
* x-pack/test/security_solution_cypress/cypress/objects/timeline.ts
* x-pack/test/security_solution_cypress/cypress/tasks/api_calls/timelines.ts
```
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This PR creates a UI component to list entities inside the Entity Store.
### What is included
- Create `EntitiesList` component
- Duplicate `MultiselectFilter` component
- Display `EntitiesList` in the entity analytics dashboard
- Use the `entityStoreEnabled` experimental flag
### What is NOT included
- Asset criticality
- Source field
- Risk score fields
### How to test
1. Add some host/user data
* Easiest is to use
[elastic/security-data-generator](https://github.com/elastic/security-documents-generator)
2. Make sure to add `entityStoreEnabled` under
`xpack.securitySolution.enableExperimental` in your `kibana.dev.yml`
3. In kibana dev tools or your terminal, call the `INIT` route for
either `user` or `host`.
4. You should now see 2 transforms in kibana. Make sure to re-trigger
them if needed so they process the documents.
5. Enable the experimental flag `entityStoreEnabled`
6. Go to entity analytics dashboard and you should see an populated
entities page
Implements https://github.com/elastic/security-team/issues/10536
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
<!--ONMERGE {"backportTargets":["8.x"]} ONMERGE-->
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
1. Create a new public API endpoint : `DELETE
/api/risk_score/engine/dangerously_delete_data`
## Test cases Result
```
PASS x-pack/plugins/security_solution/server/lib/entity_analytics/risk_engine/routes/delete.test.ts (7.017 s)
risk engine cleanup route
invokes the risk engine cleanup route
✓ should call the router with the correct route and handler (71 ms)
✓ returns a 200 when cleanup is successful (64 ms)
✓ returns a 500 when cleanup is unsuccessful (57 ms)
✓ returns a 500 when cleanup is unsuccessful with multiple errors (53 ms)
when task manager is unavailable
✓ returns a 400 when task manager is unavailable (55 ms)
when user does not have the required privileges
✓ returns a 403 when user does not have the required privileges (88 ms)
```
### API Responses
## When multiple errors encountered
```
{
"risk_engine_cleanup": false,
"errors": [
{
"seq": 1,
"error": "resource_not_found_exception\n\tRoot causes:\n\t\tresource_not_found_exception: Transform with id [risk_score_latest_transform_default] could not be found"
},
{
"seq": 2,
"error": "index_not_found_exception\n\tRoot causes:\n\t\tindex_not_found_exception: no such index [risk-score.risk-score-default]"
},
{
"seq": 3,
"error": "index_template_missing_exception\n\tRoot causes:\n\t\tindex_template_missing_exception: index_template [.risk-score.risk-score-default-index-template] missing"
},
{
"seq": 4,
"error": "resource_not_found_exception\n\tRoot causes:\n\t\tresource_not_found_exception: .risk-score-mappings"
}
],
"status_code": 500
}
```
## Success
```
{
"risk_engine_cleanup": true
}
```
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### OpenAPI spec
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
**Relates to:** https://github.com/elastic/kibana/issues/184428
## Summary
This PR omit `Solution` from from tag names and titles in Security Solution's OpenAPI bundles based on Security Docs team recommendation.
## Summary
Adds basic end to end tests for the engine CRUD routes.
I also noticed there was some inconsistency in the naming of the API
routes which I have fixed.
---------
Co-authored-by: machadoum <pablo.nevesmachado@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Tiago Vila Verde <tiago.vilaverde@elastic.co>
## Summary
Fixes: https://github.com/elastic/security-team/issues/10133
Migrates some timeline routes to use the newly generated OpenAPI types.
The changes mostly affect pinned event and note routes to keep the
changes small. Routes that actually accept and return timeline objects
will come in a next step.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Closes#188514
Adds OAS schemas for the `403 Forbidden` errors that public rule apis
can return if a license is invalid, `400 Bad Request` for unregistered
rule types, and `404 Not Found` for missing saved objects.
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
### Testing
1. Start ES
2. Add `server.oas.enabled: true` to `kibana.dev.yml`
3. Start Kibana `yarn start --no-base-path`
4. `curl -s -uelastic:changeme
http://localhost:5601/api/oas\?pathStartsWith\=/api/alerting/rule/ | jq`
(If you have `jq` installed, otherwise pipe to `pbcopy` and paste the
result into a JSON prettifier)
5. Search the output for the word `Forbidden` to ensure this schema has
been added to `create`, `update`, `enable`, `disable`, `mute`, `unmute`,
and `update_rule_api_key`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Resolves https://github.com/elastic/kibana/issues/192778
## Summary
This PR updates the following `response` schemas as well as the legacy
route schemas for connector APIs to generate OAS documentation:
- `POST /api/actions/connector/{id?}`
- `GET /api/actions/connector/{id}`
- `POST /api/actions/connector/{id}/_execute`
- `PUT /api/actions/connector/{id}`
The `request` schemas were updated in this
[PR](https://github.com/elastic/kibana/pull/191678).
### To verify
1. Start ES
2. Add `server.oas.enabled: true` to `kibana.dev.yml`
3. Start Kibana `yarn start --no-base-path`
4. `curl -s -uelastic:changeme
http://localhost:5601/api/oas\?pathStartsWith\=/api/actions/ | jq`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
