Disable entity discovery on plugin startup and remove some related
files.
We'll need a follow up to cleanup endpoints.
### Testing
- enable entity discovery from `main` with `PUT
kbn:/internal/entities/managed/enablement`, preferably in a cluster with
data so you can verify that the `.entities*` indices are also deleted
- checkout that branch and start kibana
- verify all v1 builtin definitions with their
transforms/templates/pipelines are deleted. `.entities` indices for
these definitions should also be gone
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
implements `countEntities` API
the query to count a single-source definition is straightforward but
gets tricky when sources > 1 because we have to resolve entity ids to
avoid counting duplicates. I've reused the entity.id/source eval logic
implemented here
https://github.com/elastic/elastic-entity-model/issues/202#issuecomment-2500608664
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
- small refactor of _search code which is now resilient to source-level
failure. We now return `{ entities: EntityV2[], errors: string[] }`
where `errors` lists source failures while still collecting successful
sources in `entities`
- add integration tests
- fixes some issues in the merging logic uncovered by the tests
- change metadata aggregation from `VALUES` to `TOP 10` since the former
is in tech preview
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Update built in definitions on plugin start. The update overwrites index
templates and ingest pipelines with the latest versions but has to
delete the transforms since we can only update a subset of settings in
the [update
api](https://www.elastic.co/guide/en/elasticsearch/reference/current/update-transform.html)
which does not include the aggregations.
## Testing
### api tests
Since the upgrade logic happens in plugin startup method we cannot
directly trigger it from api tests without some tweaks. I've added a
[fixture
plugin](a87ae8b210/x-pack/test/api_integration/apis/entity_manager/fixture_plugin/server/plugin.ts)
that is launched in the entity manager test server, this plugin creates
a test route exposing the upgrade api which can then be called in api
tests.
### manual
- install builtin definitions `PUT
kbn:/internal/api/entities/managed/enablement`
- bump builtin [service
definition](https://github.com/elastic/kibana/blob/main/x-pack/plugins/observability_solution/entity_manager/server/lib/entities/built_in/services.ts#L23)
version
- restart kibana server
- logs should output `[INFO ][plugins.entityManager] Updating built-in
entity definition [builtin_services] from v0.1.0 to v<new version>`
- `GET kbn:/internal/api/entities/definition` should output the new
definition
- verify latest version of definition components are installed
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
In https://github.com/elastic/kibana/pull/188410 we moved history and
latest index templates from global scope to definition scope. The
definition-scoped templates have a wide pattern that would grep any
other definition template already installed and throw the following
error because of conflicting priority. This change narrows down the
index patterns defined in the templates to only grep the ones from the
installed definition
```
{
"statusCode": 500,
"error": "Internal Server Error",
"message": """[illegal_argument_exception
Root causes:
illegal_argument_exception: index template [entities_v1_history_admin-console-services_index_template] has index patterns [.entities.v1.history.*] matching patterns from existing templates [entities_v1_history_builtin_services_from_ecs_data_index_template] with patterns (entities_v1_history_builtin_services_from_ecs_data_index_template => [.entities.v1.history.*]) that have the same priority [200], multiple index templates may not match during index creation, please use a different priority]: index template [entities_v1_history_admin-console-services_index_template] has index patterns [.entities.v1.history.*] matching patterns from existing templates [entities_v1_history_builtin_services_from_ecs_data_index_template] with patterns (entities_v1_history_builtin_services_from_ecs_data_index_template => [.entities.v1.history.*]) that have the same priority [200], multiple index templates may not match during index creation, please use a different priority"""
}
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Updates to the EEM enable/disable endpoint:
- disable endpoint was not verifying the current user permissions, it
was just pulling the api key and removing all resources if the key was
valid. We're now checking that current user can stop/delete resources
and not using the api key at all
- the enable/disable flow now use use current user (the api caller)
clients instead of clients scoped to the api key
- updated apikey calls made with the internal user instead of the
current user
- refactored the privileges file to make it clearer
- created entity manager api tests config with enablement flow tests
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
