This PR turns the `entity.id` field format from a hashed value to a
human readable string of the **values** found in the identity fields,
such as `my_host-my_cloud_zone` for the identity fields `[host.name,
cloud.availability_zone]`.
The order of the values is based on the order in the identity fields
list.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
We were trying to cleanup `.entities` indices with the system user that
does not have necessary privileges. This failed silently because of
`ignore_unavailable: true`
### Summary
Remove history and backfill transforms, leaving latest transform in
place.
Notable changes to latest transform:
- it does not read from history output anymore but source indices
defined on the definition
- it defines a `latest.lookbackPeriod` to limit the amount of data
ingested, which defaults to 24h
- each metadata aggregation now accepts a
`metadata.aggregation.lookbackPeriod` which defaults to the
`latest.lookbackPeriod`
- `entity.firstSeenTimestamp` is removed. this should be temporary until
we have a solution for
https://github.com/elastic/elastic-entity-model/issues/174
- latest metrics used to get the latest pre-computed value from history
data, but is it now aggregating over the `lookbackPeriod` in the source
indices (which can be filtered down with `metrics.filter`)
- `latest` block on the entity definition is now mandatory
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Mark Hopkin <mark.hopkin@elastic.co>
Create `PATCH kbn:/internal/entities/definition` allowing update of
stored definition. The endpoint accepts an update object representing a
partial entity definition with a few key properties that cannot be
updated. The update process will stop and delete the existing transforms
and create new ones, there's
https://github.com/elastic/elastic-entity-model/issues/136 logged as a
follow up improvement.
### Testing
- call `PUT kbn:/internal/entities/definition/{id}` with an update
payload (see [update
schema](7a7fbdf1cd/x-pack/packages/kbn-entities-schema/src/schema/entity_definition.ts (L62)))
- call `GET kbn:/internal/entities/definition` and verify the update in
reflected in the stored definition
- verify the updated properties are reflected in the corresponding
transform/pipeline components
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR adds some basic integration tests. I also had to modify the
Entity schemas to allow for parsing the dynamic metadata fields. I don't
love the type solution and would ideally like to have something like:
```typescript
const entitySchema = createEntitySchemaFromDefintion(entityDefinintion);
```
which would dynamically generate an entity schema from the definition
based on the identity and metadata fields. I have a prototype that sort
of works but still needs a lot of work and started to become a huge
"time suck".
Closes [139](https://github.com/elastic/elastic-entity-model/issues/139)
When creating a definition we automatically start the transforms when
all the components are installed. This change introduces `installOnly:
boolean` query string that can be passed to routes that create
definitions (`POST /internal/entities/definition` and `PUT
/internal/entities/managed/enablement`) in order to only install the
components
### Testing
- `PUT kbn:/internal/entities/managed/enablement?installOnly=true`
- `GET kbn:/internal/entities/definition` returns builtin definitions
with `{ state: { installed: true, running: false } }`
- check the installed transforms are not started
In https://github.com/elastic/kibana/pull/188410 we moved history and
latest index templates from global scope to definition scope. The
definition-scoped templates have a wide pattern that would grep any
other definition template already installed and throw the following
error because of conflicting priority. This change narrows down the
index patterns defined in the templates to only grep the ones from the
installed definition
```
{
"statusCode": 500,
"error": "Internal Server Error",
"message": """[illegal_argument_exception
Root causes:
illegal_argument_exception: index template [entities_v1_history_admin-console-services_index_template] has index patterns [.entities.v1.history.*] matching patterns from existing templates [entities_v1_history_builtin_services_from_ecs_data_index_template] with patterns (entities_v1_history_builtin_services_from_ecs_data_index_template => [.entities.v1.history.*]) that have the same priority [200], multiple index templates may not match during index creation, please use a different priority]: index template [entities_v1_history_admin-console-services_index_template] has index patterns [.entities.v1.history.*] matching patterns from existing templates [entities_v1_history_builtin_services_from_ecs_data_index_template] with patterns (entities_v1_history_builtin_services_from_ecs_data_index_template => [.entities.v1.history.*]) that have the same priority [200], multiple index templates may not match during index creation, please use a different priority"""
}
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
