## Summary
This consumes the security solution's API key flyout and token selector
in Serverless search to have a consistent experience across the stack.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR updates the Rollup badge in the indices list and the data views
list so that it provides deprecation information. The tooltip component
is stored in a new package to avoid cyclical dependencies as it is used
in both the Index management plugin and in the Data views management
plugin.
<details><summary>Screenshots</summary>
<img width="1679" alt="Screenshot 2024-06-17 at 18 08 35"
src="1d3daa9d-3f62-49f8-803f-1b3d4605f9a4">
<img width="1679" alt="Screenshot 2024-06-17 at 18 08 53"
src="2e88d119-88e7-4d01-bab9-bc2add82b91c">
<img width="1679" alt="Screenshot 2024-06-17 at 18 09 18"
src="29d3d3f2-de46-45ea-96a1-b83495d122df">
</details>
**How to test:**
1. Start Es and Kibana
2. Add the sample data "Sample web logs"
3. Elasticsearch only allows creating a rollup job if there is an
existing rollup usage in the cluster. To simulate rollup usage, create a
mock rollup index through Console:
```
PUT /mock_rollup_index
{
"mappings": {
"_meta": {
"_rollup": {
"id": "logs_job"
}
}
}
}
```
4. Create a sample rollup job through Console:
```
PUT _rollup/job/logs_job
{
"id": "logs_job",
"index_pattern": "kibana_sample_data_logs",
"rollup_index": "rollup_logstash",
"cron": "* * * * * ?",
"page_size": 1000,
"groups": {
"date_histogram": {
"interval": "60m",
"delay": "7d",
"time_zone": "UTC",
"field": "@timestamp"
},
"terms": {
"fields": [
"geo.src",
"machine.os.keyword"
]
},
"histogram": {
"interval": "1003",
"fields": [
"bytes",
"memory"
]
}
}
}
```
5. Delete the mock rollup index since it causes issues for the rollup
API that we use to fetch rollup indices:
`DELETE /mock_rollup_index`
6. Navigate to Index Management and toggle the "Include rollup indices"
switch
7. Verify that the rollup indices have the deprecation badge.
8. Navigate to Data Views and create a rollup data view with index
pattern that matches the created rollup index (`rollup*`).
9. In the list of data views, verify that the rollup data view has the
correct rollup deprecation badge.
10. Click on the rollup data view and verify that the details panel also
has the deprecation badge.
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [x] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [x] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [x] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/66716
Improves code editors in Anomaly detection, Data frame analytics and
Transform wizards with autocomplete, data types validation and inline
documentation from elasticsearch specification.
Adds a package with JSON schemas extracted from the [openapi output of
elasticsearch-specification](https://github.com/elastic/elasticsearch-specification/tree/main/output/openapi).
Schema file is generated per editor/endpoint, keeping only relevant
components.
To test the script locally, execute
```
yarn run jsonSchema
```
from `/x-pack/packages/ml/json_schemas`.
By default it assumes that your `elasticsearch-specification` folder is
located next to the `kibana` repo, but you can also provide a path to
`openapi` file as a parameter, e.g. `yarn run jsonSchema
/Users/my_user/dev/elasticsearch-specification/output/openapi/elasticsearch-serverless-openapi.json`
#### How JSON files are served
JSON files are asynchronously imported at the React component level and
passed to the `CodeEditor` instances on mount.
Initially I tried different approaches to take advantage of CDN, but
unfortunately it didn't work out:
##### Using static assets
I made an attempt to retrieve a URL to the JSON schema file as a static
asset using Kibana service
```typescript
const schemaJsonAsset = http?.staticAssets.getPluginAssetHref('my_schema.json') ?? '';
```
and passing it as part of the schema definition, but the browser was
blocking a request 🤔
##### Using raw loader
Approach with a raw loader also didn't succeed.
```typescript
import mySchema from '!!raw-loader!./my_schema.json';
```
<details>
<summary>See error </summary>
```
ERROR in
./public/app/sections/create_transform/components/advanced_pivot_editor/my_schema.json
(/Users/dimaarnautov/Repos/kibana/node_modules/raw-loader/dist/cjs.js!./public/app/sections/create_transform/components/advanced_pivot_editor/my_schema.json)
│ Module parse failed: Unexpected token 'e', "export def"... is not
valid JSON while parsing 'export default "{\n \"type\": \"object\'
│ File was processed with these loaders:
│ * ../../../node_modules/raw-loader/dist/cjs.js
│ You may need an additional loader to handle the result of these
loaders.
│ SyntaxError: Unexpected token 'e', "export def"... is not valid JSON
while parsing 'export default "{\n \"type\": \"object\'
│ at JSON.parse (<anonymous>)
│ at parseJson
(/Users/dimaarnautov/Repos/kibana/node_modules/json-parse-better-errors/index.js:7:17)
│ at JsonParser.parse
(/Users/dimaarnautov/Repos/kibana/node_modules/webpack/lib/JsonParser.js:16:16)
│ at
/Users/dimaarnautov/Repos/kibana/node_modules/webpack/lib/NormalModule.js:482:32
│ at
/Users/dimaarnautov/Repos/kibana/node_modules/webpack/lib/NormalModule.js:358:12
│ at
/Users/dimaarnautov/Repos/kibana/node_modules/loader-runner/lib/LoaderRunner.js:373:3
│ at iterateNormalLoaders
(/Users/dimaarnautov/Repos/kibana/node_modules/loader-runner/lib/LoaderRunner.js:214:10)
│ at iterateNormalLoaders
(/Users/dimaarnautov/Repos/kibana/node_modules/loader-runner/lib/LoaderRunner.js:221:10)
│ at
/Users/dimaarnautov/Repos/kibana/node_modules/loader-runner/lib/LoaderRunner.js:236:3
│ at runSyncOrAsync
(/Users/dimaarnautov/Repos/kibana/node_modules/loader-runner/lib/LoaderRunner.js:130:11)
│ at iterateNormalLoaders
(/Users/dimaarnautov/Repos/kibana/node_modules/loader-runner/lib/LoaderRunner.js:232:2)
│ at
/Users/dimaarnautov/Repos/kibana/node_modules/loader-runner/lib/LoaderRunner.js:205:4
│ at
/Users/dimaarnautov/Repos/kibana/node_modules/webpack/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:85:15
│ at processTicksAndRejections (node:internal/process/task_queues:77:11)
```
</details>
### Checklist
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
This PR adds client hooks and basic REST API's for accessing and
mutating Knowledge Base Entries. This is in support of @angorayc
building out the new Knowledge Base settings interface.
Change set includes:
- [X] Refactors existing KB client hooks from
`x-pack/packages/kbn-elastic-assistant/impl/knowledge_base` to be
co-located next to the API methods where we put all our other hooks:
`x-pack/packages/kbn-elastic-assistant/impl/assistant/api/knowledge_base`
- [X] Refactors existing KB API calls and associated tests out of
`kbn-elastic-assistant/impl/assistant/api/index.tsx` and into
`x-pack/packages/kbn-elastic-assistant/impl/assistant/api/knowledge_base/api.tsx`
- [X] Adds new `find_knowledge_base_entries_route.schema.yaml` OAS for
the supporting
`/internal/elastic_assistant/knowledge_base/entries/_find` route
- [X] Refactors `SortOrder` out of existing OAS's into the shared
`schemas/common_attributes.schema.yaml`
### Client Hooks & Routes
Adds new `useKnowledgeBaseEntries()` hook and corresponding
`/knowledge_base/entries/_find` route for returning paginated KB Entries
to populate the KB table in settings. E.g.
``` ts
const {
assistantFeatures: { assistantKnowledgeBaseByDefault: enableKnowledgeBaseByDefault },
http,
toasts,
} = useAssistantContext();
const { data: kbEntries, isLoading: isLoadingEntries } = useKnowledgeBaseEntries({ http });
```
###### Sample Response
``` json
{
"perPage": 20,
"page": 1,
"total": 145,
"data": [
{
"timestamp": "2024-06-05T21:19:56.482Z",
"id": "CtBF6o8BSQy1Bdxt2FHz",
"createdAt": "2024-06-05T21:19:56.482Z",
"createdBy": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0",
"updatedAt": "2024-06-05T21:19:56.482Z",
"updatedBy": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0",
"users": [
{
"id": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0",
"name": "elastic"
}
],
"metadata": {
"kbResource": "security_labs",
"source": "/Users/garrettspong/dev/kibana-main/x-pack/plugins/elastic_assistant/server/knowledge_base/security_labs/2022_elastic_global_threat_report_announcement.mdx",
"required": false
},
"namespace": "default",
"text": "[Source Content Here]",
"vector": {
"modelId": ".elser_model_2",
"tokens": {
"2": 0.06595266,
...
}
}
},
...
]
}
```
Response is the full newly created `entry`. Same format for the entry as
above in the `_find` API, and the `KnowledgeBaseEntries` cache is
invalidated.
Adds new `useCreateKnowledgeBaseEntry()` hook and corresponding
`/knowledge_base/entries` route for creating new KB Entries
``` ts
const entry: KnowledgeBaseEntryCreateProps = {
metadata: {
kbResource: 'user',
required: true,
source: 'user',
},
text: 'Useful information about the user',
};
const { mutate: createEntry, isLoading: isCreatingEntry } = useCreateKnowledgeBaseEntry({
http,
});
await createEntry(entry);
```
Adds new `useDeleteKnowledgeBaseEntries()` hook and corresponding
`/knowledge_base/entries/_bulk_action` route for deleting existing KB
Entries. I left a TODO to plumb through `delete_by_query` so we can add
a filter bar to the table. Need to confirm if we can do pagination with
similarity search as well.
``` ts
const { mutate: deleteEntries, isLoading: isDeletingEntries } = useDeleteKnowledgeBaseEntries({
http,
});
await deleteEntries({ ids: ['YOE_CZABSQy1BdxtAGbs'] })
```
See `KnowledgeBaseEntryBulkCrudActionResponse` for response formats.
`KnowledgeBaseEntries` cache is invalidated upon delete.
### Checklist
Delete any items that are not applicable to this PR.
- [ ] ~Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)~
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
* Feature currently behind feature flag. Documentation to be added
before flag is removed. Tracked in
https://github.com/elastic/security-docs/issues/5337
- [X] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] API tests will need to be rounded out as we finalize functionality
behind the feature flag
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
If we analyse all detected text fields, we might run into performance
issues with the `categorize_text` aggregation. Until this is resolved,
we will rely on a predefined white list of supported text fields, for
now set to `message` and `error.message`.
Currently, when a new inference endpoint is created from inference
flyout, the flyout stays open until the endpoint is created. This may
take long time when Elasticsearch models - `.elser_model_2` and
`.multilingual-e5-small` is to be downloaded, deployed and inference
endpoint is to be created.
In this PR, When a new inference endpoint is saved, inference flyout is
closed and the new inference endpoint is created by a callback function
in the component.
**Screen Recording**
8eabba1a-108a-4bf2-813a-66ceb291467c
**Testing instructions**
**update Elasticsearch to latest (only to test save mappings)**
Since ES changes for the semantic_text has been merged to main, this can
be tested against running ES from source or from latest snapshot
- Update local branch with latest Elasticsearch changes from main
- Run the elasticsearch: ./gradlew :run -Drun.license_type=trial
- Manual test in UI
**Frontend**
- enable` xpack.index_management.dev.enableSemanticText` to true in
`config/kibana.dev.yml`
- Add a new field with type - Semantic_text
- Click on drop down menu below `Select an inference endpoint`
- Click Add inference Endpoint
- Type new inference endpoint name and click Save endpoint
- Save endpoint button should close the flyout
- A new success notification toasts is shown with text "1 model is being
deployed on your ml_node."
- Add new field
- Click Save mappings
- should show a modal with model deployment status
- After new endpoint is created, refresh button should hide the modal
and save mappings should update mappings
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Migrates our existing RAG pipeline to use LangGraph, and adds tools for
Knowledge Base retrieval/storage.
When the `assistantKnowledgeBaseByDefault` FF is enabled, a new branch,
`callAssistantGraph()`, is taken in `postActionsConnectorExecuteRoute`
that exercises the LangGraph implementation. This is a drop-in
replacement for the existing `callAgentExecutor()` in effort to keep
adoption as clean and easy as possible.
The new control flow is as follows:
`postActionsConnectorExecuteRoute` -> `callAssistantGraph()` ->
`getDefaultAssistantGraph()` -> `isStreamingEnabled ? streamGraph() :
invokeGraph()`
Graph creation is isolated to `getDefaultAssistantGraph()`, and
execution (streaming or not) has been extracted to `streamGraph()` and
`invokeGraph()` respectively. Note: Streaming currently only works with
`ChatOpenAI` models, but `SimpleChatModelStreaming` was de-risked and
just need to discuss potential solutions with @stephmilovic. See
[comment
here](https://github.com/elastic/kibana/pull/184554/files#diff-ad87c5621b231a40810419fc1e56f28aeb4f8328e125e465dfe95ae0e1c305b8R97-R98).
#### DefaultAssistantGraph
To start with a predictable and piecemeal migration, our existing
`agentExecutor` pipeline has been recreated in LangGraph. It consists of
a single agent node, either `OpenAIFunctionsAgent`, or
`StructuredChatAgent` (depending on the backing LLM), a tool executing
node, and a conditional edge that routes between the two nodes until
there are no more function calls chosen by the agent. This varies from
our initial implementation in that multiple tool calls are now
supported, so a user could ask about their alerts AND retrieve
additional knowledge base information in the same response.
> [!NOTE]
> While `chat_history` has been plumbed into the graph, after discussing
with @YulNaumenko we decided to wait to plumb the rest of persistence
into the graph until #184485 is merged. I had already plumbed through
the `chatTitleGeneration` node
([here](https://github.com/elastic/kibana/pull/184554/files#diff-26038489e9a3f1a14c5ea2ac2954671973d833349ef3ffaddcf9b29ce9e2b96eR33)),
and so will just need to include initial conversation creation and
append/update operations.
#### Knowledge History & KB Tools
Knowledge History is now always added in the initial prompt for any KB
documents marked as `required`, and two new tools were added for
creating and recalling KB entries from within the conversation,
`KnowledgeBaseWriteTool` and `KnowledgeBaseRetrievalTool` respectively.
All three methods of storing and retrieving KB content use the
`kbDataClient` for access, and scopes all requests to the
authenticatedUser that made the initial request.
Additional Notes:
* LangChain dependencies have been updated, and a new dependency on
`LangGraph` has been added.
### Checklist
Delete any items that are not applicable to this PR.
- [X] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
* Feature currently behind a FF, documentation to be added once feature
is complete. Tracked in
https://github.com/elastic/security-docs/issues/5337.
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
* Test coverage in progress...
---------
Co-authored-by: Patryk Kopycinski <contact@patrykkopycinski.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This is a PR to add a new backend plugin (frontend will be done in
separate [PR](https://github.com/elastic/kibana/pull/184546)).
The purpose of the plugin is to provide a set of API routes that is used
to perform a variety of GenAI workflows to generate new integrations
based on provided inputs.
It reuses the existing GenAI connectors for its LLM communication, and
provides a set of API's to create ECS mapping, Categorization, Related
Fields and an API to generate the actual integration package zip, which
is forwarded to the UI component.
### Planned follow-up changes:
As the PR is getting way too large, some planned changes would be added
in much smaller follow-ups. This includes mostly more improved try/catch
for certain routes, adding debug/error log entries where relevant,
especially for the API endpoints themself, some more unit and end2end
tests.
- OpenAPI spec for the API will be handled in a separate PR
- All the missing unit tests will be added as a followup PR
### Testing
The `integration_assistant` plugin will be disabled by default while
it's being implemented so we can iterate and merge partial PRs without
interfering with the releases. This config will work as our feature
flag:
6aefd4ff7b/x-pack/plugins/integration_assistant/server/config.ts (L11-L13)
To test it add this to your _kibana.dev.yml_:
```
xpack.integration_assistant.enabled: true
```
### Checklist
Delete any items that are not applicable to this PR.
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: Patryk Kopycinski <contact@patrykkopycinski.com>
Co-authored-by: Sergi Massaneda <sergi.massaneda@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Bharat Pasupula <saibharatchandra.pasupula@elastic.co>
Co-authored-by: Bharat Pasupula <123897612+bhapas@users.noreply.github.com>
Addresses #184751
## Summary
This PR addresses couple of issues:
### Main:
Persist revamped `resultsFieldMap` schema fields, namely
`incompatibleFieldMappingItems`, `incompatibleFieldValueItems` and
`sameFamilyFieldItems` in the `StorageResult` after index check, so that
after release user can start accumulating data in these fields, while we
prepare main UI changes.
### Additional:
Improve and narrow down existing in-house `EcsFlat` override type that
originally comes from `@elastic/ecs` npm package, because currently it
is too generic and too loose, resulting in an unnecessary conditional
checks and leads to perception of impossible states most of which are
refactored, cleaned and fixed in this PR.
### Screenshots
### How to test
1. Prepare index with invalid mapping and value fields + 1 same family
field
```graphql
DELETE test-field-items
PUT test-field-items
{
"mappings": {
"properties": {
"event.category": { "type": "keyword"},
"agent.type": {"type": "constant_keyword" },
"source.ip": {"type": "text"}
}
}
}
PUT test-field-items/_doc/1
{
"@timestamp": "2016-05-23T08:05:34.853Z",
"event.category": "behavior"
}
PUT test-field-items/_doc/2
{
"@timestamp": "2016-05-23T08:05:34.853Z",
"event.category": "shmehavior"
}
```
2. Open DQD dashboard in kibana
3. Create `test-*` data-view with `test-*` index pattern
4. Select it in the sourcerer
5. Click expand button near test-field-items index
6. Verify that you have 1 mapping + 1 value incompatible field + 1 same
family field
7. Open kibana devtools
8. Run
```graphql
GET .kibana-data-quality-dashboard-results-default/_search
{
"size": 0,
"query": {
"term": {
"indexName": {
"value": "test-field-items"
}
}
},
"aggs": {
"latest": {
"terms": { "field": "indexName", "size": 10000 },
"aggs": {
"latest_doc": {
"top_hits": {
"size": 1,
"sort": [{ "@timestamp": { "order": "desc" } }]
}
}
}
}
}
}
```
9. Verify that latest result contains `incompatibleFieldItems` and
`sameFamilyFieldItems` of expected shape:
```json5
//...
"incompatibleFieldValueItems": [
{
"fieldName": "event.category",
"expectedValues": [
"api",
"authentication",
"configuration",
"database",
"driver",
"email",
"file",
"host",
"iam",
"intrusion_detection",
"library",
"malware",
"network",
"package",
"process",
"registry",
"session",
"threat",
"vulnerability",
"web"
],
"actualValues": [
{ "name": "behavior", count: 2 },
{ "name": "shmehavior", count: 1}
],
"description": """This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy.
`event.category` represents the "big buckets" of ECS categories. For example, filtering on `event.category:process` yields all events relating to process activity. This field is closely related to `event.type`, which is used as a subcategory.
This field is an array. This will allow proper categorization of some events that fall in multiple categories."""
}
],
"incompatibleFieldMappingItems": [
{
"fieldName": "source.ip",
"expectedValue": "ip",
"actualValue": "text",
"description": "IP address of the source (IPv4 or IPv6)."
}
]
//...
"sameFamilyFieldItems": [
{
"fieldName": "agent.type",
"expectedValue": "keyword",
"actualValue": "constant_keyword",
"description": """Type of the agent.
The agent type always stays the same and should be given by the agent used. In case of Filebeat the agent would always be Filebeat also if two Filebeat instances are run on the same machine."""
}
]
```
## Summary
This PR moves disclaimer comment section to the top of the generated
files to support `import/order` ESlint rule.
## Details
`kbn-openapi-generator` generates `<schema-name>.gen.ts` files for each
encountered schema with enabled code generation. The generate file
contains imports of the referenced schema from the other generated
files. Everything works until there is a reference to a package or
another plugin. Consider an example below where we have a generated file
with an import from `kbn-openapi-common` package (this package doesn't
exist in reality and used for clarity but the same can be shown with
cross plugin references)
```ts
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { z } from 'zod';
/*
* NOTICE: Do not edit this file manually.
* This file is automatically generated by the OpenAPI Generator, @kbn/openapi-generator.
*
* info:
* title: Shared Alert Primitives Schema
* version: not applicable
*/
import { NonEmptyString } from '@kbn/openapi-common/primitives';
/**
* A list of alerts ids.
*/
export type AlertIds = z.infer<typeof AlertIds>;
export const AlertIds = z.array(NonEmptyString).min(1);
```
If `import/order` rule is enabled for this file linting with fixing will
fail with an error `8:1 error There should be no empty line within
import group import/order` since auto-fix can't fix the file due to the
comment between imports. Linting with auto-fixing is a part of code
generation process which means code generation will fail in that case.
For example lists plugin has `import/order` rule enabled.
The problem is fixed by moving disclaimer (NOTICE) section to the top
just right above the first import. Since the whole file is
auto-generated it makes sense.
## Summary
Fixes https://github.com/elastic/kibana/issues/173771
Use saved data view id instead of index pattern where it's available.
Inject runtime mappings from the dataview into transform.
- [ ] Go to Discover and add a runtime field to the data view (this is
only available in Discover)
- [ ] Make sure filtering works based on the data view
We are not supporting "scripted fields" from the Index Management
DataView editor.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR closes https://github.com/elastic/observed-asset-model/issues/61
by adding a second transform to generate a history of the entities into
`.entities-observability.history-v1.{definition.id}.{YYYY-MM-DD}`
indices. This PR also modifieds the summary transform to use the
historical data as it's source.
### Changes
- Added a section for `history` to the defintion configure the
`interval`, `timestamp`, `intialLookback`, and `settings` for the
history transform
- Added a section for `summary` to the definition to configure the
`settings` for the summary transform
- Updates the create route to create and install both the summary and
history transforms
- Updates the create route to create and install both the summary and
history ingest pipeline
- Updates the delete route to stop and delete both the summary and
history transforms
- Updates the delete route to delete both the summary and history ingest
pipelines
- Updates the reset route to remove all the history and summary indices
- Removes `indexPatterns` from the data output for both the summary and
history entities
- Renames `entity.metirc` to `entity.metrics`
- Renames `entity.identity` to `entity.identityFields`
- Modifiy `entity.id` to be a `MurmurHash3` of all the values of the
`entity.identityFields`
- Adds `entity.displayName` which uses `displayNameTemplate` to create
the string (was `entity.id`)
## Summary
Fixes an issue with auto title functionality while using Bedrock
connector, the issue was in using `assistant` role, which is not
supported as first message
```
[2024-05-22T16:04:55.149+00:00][ERROR][plugins.actions.bedrock] Error: Status code: 400. Message: API Error: Bad Request - messages: first message must use the "user" role
at BedrockConnector.request (sub_action_connector.ts:198:15)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at BedrockConnector.runApiLatest (bedrock.ts:195:22)
at BedrockConnector.invokeAI (bedrock.ts:311:17)
at Object.executor (executor.ts:87:18)
at action_executor.ts:492:23
at ActionExecutor.execute (action_executor.ts:147:12)
at executeAction (executor.ts:55:24)
at Object.fn (post_actions_connector_execute.ts:196:36)
at core_versioned_route.ts:189:22
at Router.handle (router.ts:267:30)
at handler (router.ts:195:13)
at exports.Manager.execute (/Users/patrykkopycinski/Projects/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at Object.internals.handler (/Users/patrykkopycinski/Projects/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
at exports.execute (/Users/patrykkopycinski/Projects/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
at Request._lifecycle (/Users/patrykkopycinski/Projects/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/Users/patrykkopycinski/Projects/kibana/node_modules/@hapi/hapi/lib/request.js:281:9) {"tags":[".bedrock","bedrock","action-run-failed"],"error":{"stack_trace":"Error: Status code: 400. Message: API Error: Bad Request - messages: first message must use the \"user\" role\n at BedrockConnector.request (sub_action_connector.ts:198:15)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at BedrockConnector.runApiLatest (bedrock.ts:195:22)\n at BedrockConnector.invokeAI (bedrock.ts:311:17)\n at Object.executor (executor.ts:87:18)\n at action_executor.ts:492:23\n at ActionExecutor.execute (action_executor.ts:147:12)\n at executeAction (executor.ts:55:24)\n at Object.fn (post_actions_connector_execute.ts:196:36)\n at core_versioned_route.ts:189:22\n at Router.handle (router.ts:267:30)\n at handler (router.ts:195:13)\n at exports.Manager.execute (/Users/patrykkopycinski/Projects/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n at Object.internals.handler (/Users/patrykkopycinski/Projects/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)\n at exports.execute (/Users/patrykkopycinski/Projects/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)\n at Request._lifecycle (/Users/patrykkopycinski/Projects/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)\n at Request._execute (/Users/patrykkopycinski/Projects/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)"},"service":{"node":{"roles":["background_tasks","ui"]}}}
```
Fixes#183625Fixes#183130
## Summary
This PR fixes#183130 by removing `Last 1 minute` subtitle in the alert
details pages of the metric and custom threshold rules.
|Before|After|
|---|---|
||
Also, it adjusts the chart time range by extending it at least 20 times
the rule execution interval to avoid showing no data for bigger
intervals such as 1 hour or 1 day.
## Summary
Upgrading intl packages from v2 to v6
### Packages upgrade:
- [x] Add @formatJS packages
- [x] `react-intl` Upgraded
- [x] `intl-messageformat` Upgraded
- [x] `intl-format-cache` removed
- [x] `intl-relativeformat` removed
- [x] `intl-messageformat-parser` removed
### Todo list:
- [x] Refactor HTML tags
- [x] Refactor Upgrade tags
- [x] Refactor `kbn-i18n`
- [x] Refactor `kbn-i18n-react`
- [x] Refactor `FormattedRelative` to `FormattedRelativeTime`
- [x] Refactor polyfills
- [x] Refactor IntlShape types
- [x] Rewrite Providers
- [x] Rewrite tests using i18n
- [x] Removed current pseudolocale implementation (tracker:
https://github.com/elastic/kibana/issues/180244)
- [x] Fix jest tests using rendered `Provider`
- [x] Remove no longer valid i18n packages documentation (tracker:
https://github.com/elastic/kibana/issues/180259)
Closes https://github.com/elastic/kibana/issues/178968
Closes https://github.com/elastic/kibana/issues/38642
## Notes to code reviewers
For team other than the core team, please review your plugins code
changes by filtering files by codeowners.
### Test Snapshot updates
Most of the changes are refactors of renamed functions and changed ICU
syntax.
The main updates are snapshot changes where `FormattedMessage` is now
memoized so snapshots capturing the html tree needed to be updated to
use `<Memo(MemoizedFormattedMessage)` instead of `<FormattedMessage`
### ICU now supports HTML tags:
before:
```
<FormattedMessage
defaultMessage="To buy a shoe, { link } and { cta }"
values={{
link: (
<a class="external_link" target="_blank" href="https://www.shoe.com/">
visit our website
</a>
),
cta: <strong class="important">eat a shoe</strong>,
}}
/>
```
after:
```
<FormattedMessage
defaultMessage="To buy a shoe, <a>visit our website</a> and <cta>eat a shoe</cta>"
values={{
a: msg => (
<a class="external_link" target="_blank" href="https://www.shoe.com/">
{msg}
</a>
),
cta: msg => <strong class="important">{msg}</strong>,
}}
/>
```
### Escape character to prevent ICU parsing changed from double slashes
to single quotes:
before: `\\{escaped\\}`
after: `'{escaped}'`
### No need for Intl Shape
the new packages under formatJS are written in typescript and come with
types support out of the box so no need to set types when using i18n.
Renamed `InjectedIntlProps` with `WrappedComponentProps`.
Removed `prop-types` and `intlShape` in favor of `IntlShape`.
### FormattedRelative has been renamed to FormattedRelativeTime and its
API has changed significantly. See
[FormattedRelativeTime](https://formatjs.io/docs/react-intl/upgrade-guide-3x#formattedrelativetime)
for more details.
### All tags specified must have corresponding values and will throw
error if it's missing
All tags are now parsed and expected to be formatted properly (all
opened tags must be closed).
To skip this check you can use the `ignoreTag: true` property
```
i18n.translate('xpack.apm.agentConfig.captureJmxMetrics.description', {
defaultMessage: 'This is not an HTML tag <JMX object name pattern>' +
ignoreTag: true,
}),
```
**When do I use ignore tags?**
If your message has HTML tags, it is preferred not to ignore the Tag to
have some string verification that the html tags you are adding are
properly formatted and closed.
If it the text between brackets is not an HTML tag and it is just a
fomat preference then using `ignoreTag` makes sense.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
## Summary
This PR closes https://github.com/elastic/observed-asset-model/issues/67
by capturing the `spaceId` from the API request and storing the
`entity.spaceId` via the ingest pipeline.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Address #183698
This MR introduces a new internal DQD `GET` endpoint for fetching index
results by pattern without aggregation in descending order by
`@timestamp`:
`GET /internal/ecs_data_quality_dashboard/results/:pattern`
This new endpoint supports additional query params like:
- `from` - es search pagination offset
- `size`- es search query response size
- `outcome` - es search query response filter by presence/absence of
`incompatibleFieldCount` field
- `startDate` - es search query response filter by `@timestamp`
(inclusive)
- `endDate`- es search query response filter by `@timestamp` (inclusive)
This endpoint returns a list of historical index `results` sorted by
`@timestamp` and a `total` number of results for facilitating pagination
on client.
Additionally this MR updates latest results endpoint to have simplified
naming for improved consistency, lifts shared functionality into a
helper function and adds necessary unit tests.
In particular:
- `GET
/internal/ecs_data_quality_dashboard/results/indices_latest/:pattern`
has been simplified to just `GET
/internal/ecs_data_quality_dashboard/results_latest/:pattern`. All
respective server and client code has been updated.
## How to test
Example:
- `GET
/internal/ecs_data_quality_dashboard/results/logs-*?from=5&size=5&outcome=fail&startDate=now-1w/w&endDate=now`
- `GET /internal/ecs_data_quality_dashboard/results_latest/logs-*`
## Summary
It fixes#179633
Observability created a Comparator type/enum, when ResponseOps is
already exporting one and other rules using it.
The only difference is the wording of not in between [I put the two
types side by side to compare]
Currently, we import the one in triggers-actions-ui-plugin , and then
update the not in between to match our Comparator.
### Comparing the two enums:
## For reviewers 🧪
- Everything should work as expected: Alert flyout, Alert reason
message, Rule creation flyout, etc.
- I kept the `outside` comparator (replaced by `NOT BETWEEN`) for
backward compatibility
Adds a `managed` flag to entity definition to determine if it was
created by user or by system
### Testing
- Create a definition and optionally pass a top level `managed: boolean`
flag to it
- Verify managed flag is appropriately shown in the response
```
POST kbn:/api/entities/definition
{
"id": "admin-console-logs-service",
"name": "Services for Admin Console",
"type": "service",
"indexPatterns": ["kbn-data-forge-fake_stack.*"],
"timestampField": "@timestamp",
"lookback": "5m",
"identityFields": ["log.logger"],
"identityTemplate": "{{log.logger}}",
"metadata": [
"tags",
"host.name"
],
"metrics": [
{
"name": "logRate",
"equation": "A / 5",
"metrics": [
{
"name": "A",
"aggregation": "doc_count",
"filter": "log.level: *"
}
]
},
{
"name": "errorRate",
"equation": "A / 5",
"metrics": [
{
"name": "A",
"aggregation": "doc_count",
"filter": "log.level: \"ERROR\""
}
]
}
]
}
```
## Summary
Follow up to #132590.
Part of #181111.
This updates the developer examples for `@kbn/ml-response-stream` to
include a variant with a full Redux Toolkit setup. For this case, the
`@kbn/ml-response-stream` now includes a generic slice `streamSlice`
that can be used. This allows the actions created to be streamed via
NDJSON to be shared across server and client.
Functional tests for the examples were added too. To run these tests you
can use the following commands:
```
# Start the test server (can continue running)
node scripts/functional_tests_server.js --config test/examples/config.js
# Start a test run
node scripts/functional_test_runner.js --config test/examples/config.js
```
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Summary
### Before
Serverless: The tab is hidden
ESS: We displayed the unauthorized banner
### After
Display the upsell banner
## How to test it
* For every license and tier
* Go to host/user page inside, explore menu item
* Verify what is displayed inside the risk score tab
* Run security serverless with "Security Analytics Essentials" tier
* You should see the upsell component
* Run security serverless with "Security Analytics Complete" tier
* You should NOT see the upsell component
* Run kibana ESS with platinum license
* You should NOT see the upsell component
* Run kibana ESS with basic license
* You should see the upsell component
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
Closes https://github.com/elastic/kibana/issues/178534
Replaces the pattern analysis flyout in Discover with a tab which sits
alongside the Documents and Field statistics tabs.
**Field selection**
Lists all of the text fields in the index. Auto selects `message`, then
`error.message`, then `event.original` and if none of these fields are
available, it just selects the first field in the list.
The Options menu provides some configuration options:
**Minimum time range**
Sets the minimum time range used for the pattern analysis search. The
pattern matching results results will be more accurate the more data it
sees, so if the user has selected e.g. last 15mins in the time picker,
this settings will ensure a wider time range is used to improve the
accuracy of the patterns. If the time picker has a larger time range
than this setting, the larger time range will be used.
**Random sampling**
Improves the search performance by using a random sampler. This is the
same setting as before.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
closes#183682
## Summary
The validation schema in Kibana's API key endpoints for cross cluster
API keys was missing the optional query, field_security, and
allow_restricted_indices fields. These have been added, and the schemas
have been unified between the create and update endpoints.
### Testing
Updated API integration tests to include checking create and update for
cross cluster API keys that contain all search options.
- x-pack/test/api_integration/apis/security/api_keys.ts
## Release note
Fixes an issue in Kibana cross cluster API key endpoints which kept
users from creating cross cluster API keys with all possible search
options.
## Summary
Part of #181111.
The refresh button wasn't working as expected. It would refetch the date
histogram, but if you had a time set like `Last 15 minutes`, the time
range on the page wasn't updated. This PR adds a fix to trigger a
refresh of active bounds. It fixes a problem with the deviation brush
not properly updating too. Without the fix, the deviation brush would
not move when the time range changes.
[aiops-lra-refresh-fix-0001.webm](70d51b1c-e831-4971-b385-3c455632b8eb)
### Checklist
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Summary
Part of #181111.
This filters field/value items from the results if the field has a
cardinality of just 1 since it wouldn't be useful as a result.
Before (you can easily spot fields with cardinality of 1 because the
mini histogram in the table is just one color which means the value is
present in all logs):
After:
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Summary
This PR is `Phase 1` of the Knowledge Base work for `8.15`, which
includes [automatically setting up the Knowledge
base](https://github.com/elastic/security-team/issues/9305) (this PR),
introducing new generic KB tools for recall/retrieval, a CRUD API for
for managing Knowledge Base Entries, and a basic UI for Knowledge Base
Entry management (all captured in [this issue](url)). Once complete,
this will also provide the opportunity to remove the
`!isEnabledKnowledgeBase` code paths, directing all interactions through
our LangChain Agent pipeline.
This PR sets the ground work for all of the above by moving ELSER setup
and Knowledge Base data management to use the new `AssistantDataClient`
architecture used for Conversations, AnonymizationFields and Prompts.
This feature is currently behind the `assistantKnowledgeBaseByDefault`
experimental feature flag, which can be enabled by adding the following
to your `kibana.dev.yml`:
```
xpack.securitySolution.enableExperimental:
- 'assistantKnowledgeBaseByDefault'
```
Once enabled, an `Install Knowledge Base` button will be shown when
starting a new conversation. Note: UX is still under development.
<p align="center">
<img width="700"
src="5e2397b8-a9ba-468e-9768-6b9f50b1bb33"
/>
</p>
#### Useful Dev Tools Queries
The new `assistantKnowledgeBaseByDefault` flows are quite resilient, and
so everything should function as expected even if one piece of the
puzzle is missing or incomplete. Here are some dev tool queries to check
and delete individual resources, which is nice for testing. For
instance, you can nuke the ingest pipeline, or ELSER, and the `Install
KB` button will appear and function as intended.
> [!NOTE]
> Since the existing API's were used, with forked logic for the
`assistantKnowledgeBaseByDefault` FF, the existing KB Settings UI still
functions as expected, and can be used for deleting and re-initializing
the KB. This functionality will most likely go away with updates to the
KB UI, but is nice for testing in the interim.
<details><summary>Useful Dev Tools Queries</summary>
<p>
``` ts
// New KB
GET /_ingest/pipeline/.kibana-elastic-ai-assistant-ingest-pipeline-knowledge-base
GET /_index_template/.kibana-elastic-ai-assistant-index-template-knowledge-base
GET /_data_stream/.kibana-elastic-ai-assistant-knowledge-base-default/
GET .kibana-elastic-ai-assistant-knowledge-base-default/_count
GET .kibana-elastic-ai-assistant-knowledge-base-default/_mapping
GET .kibana-elastic-ai-assistant-knowledge-base-default/_search
{ "size": 1000 }
// MSearch for if ES|QLKB docs exists
GET .kibana-elastic-ai-assistant-knowledge-base-default/_msearch
{}
{"query":{"bool":{"must_not":[{"term":{"metadata.kbResource":"esql"}},{"term":{"metadata.required":true}}],"must":[{"text_expansion":{"vector.tokens":{"model_id":".elser_model_2","model_text":"You can chain processing commands, separated by a pipe character: `|`."}}}]}},"size":10}
{}
{"query":{"bool":{"must":[{"term":{"metadata.kbResource":"esql"}},{"term":{"metadata.required":true}}]}},"size":10000}
// Other DataClient Assets
GET .kibana-elastic-ai-assistant-anonymization-fields-default/_search
{ "size": 1000 }
GET .kibana-elastic-ai-assistant-conversations-default/_search
{ "size": 1000 }
GET .kibana-elastic-ai-assistant-prompts-default/_search
{ "size": 1000 }
GET /_ingest/pipeline/.kibana-elastic-ai-assistant-ingest-pipeline-knowledge-base
// Delete them all!
// Data Streams
DELETE /_data_stream/.kibana-elastic-ai-assistant-anonymization-fields-default
DELETE /_data_stream/.kibana-elastic-ai-assistant-conversations-default
DELETE /_data_stream/.kibana-elastic-ai-assistant-knowledge-base-default
DELETE /_data_stream/.kibana-elastic-ai-assistant-prompts-default
// Index Templates
DELETE /_index_template/.kibana-elastic-ai-assistant-index-template-anonymization-fields
DELETE /_index_template/.kibana-elastic-ai-assistant-index-template-conversations
DELETE /_index_template/.kibana-elastic-ai-assistant-index-template-knowledge-base
DELETE /_index_template/.kibana-elastic-ai-assistant-index-template-prompts
// Pipelines
DELETE /_ingest/pipeline/.kibana-elastic-ai-assistant-ingest-pipeline-knowledge-base
```
</p>
</details>
##### New Features:
- [X] Plumbed through new `assistantKnowledgeBaseByDefault` experimental
feature flag and exposed through `assistantCapabilities` API
- [X] Cleaned up `assistantFeatures` made available in AssistantProvider
and tests (no need to individually plumb new features)
- [X] Introduced new `AIAssistantDataClient` for creating Data Streams,
Component Templates, and Ingest Pipeline
- [X] Use `AIAssistantDataClient` to automatically install ELSER via
`installElasticModel()` API, then deploy via TrainedModelsAPI
- [X] Plumb through `addKnowledgeBaseDocuments()` for creating KB
entries from LangChain `Documents` within `AIAssistantDataClient`
- [X] Update `ElasticsearchStore` to take a `kbDataClient` for use in
document adding/retrieval
##### Changes not behind FF:
- [X] Updated `getELSER()` helper function to be called by
`internalUser` to prevent ml privilege requirements to setup Knowledge
Base once a privileged user has enabled ELSER
- [X] Updated `get/post/delete` knowledge base routes to create
`esStore` as `internalUser` to enable all assistant users the ability to
enable the KB once ELSER has been installed (currently they `503` if
currentUser doesn't have `read_ingest` and `manage`/`read` for the
`.kibana-elastic-ai-assistant-kb index`)
- [X] Updated `get/post/delete` knowledge base routes with assistant API
access controls: `tags: ['access:elasticAssistant'],`
- [X] Relaxed id validation from UUID to UUID or NonEmptyString to
support ES generated id's, see:
[ddf93a8](ddf93a8dd1)
### Checklist
Delete any items that are not applicable to this PR.
- [X] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- TBD once FF is removed
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- Existing tests updated with new functionality, new tests to be fleshed
out as feature stabilizes before removing FF
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This is a "proof of concept" for generating entity-centric indices for
the OAM. This exposes an API (`/api/entities`) for creating "asset
definitions" (`EntityDefinition`) that manages a transform and ingest
pipeline to produce documents into an index which could be used to
create a search experience or lookups for different services.
### Features
- Data schema agnostic, works with known schemas OR custom logs
- Supports defining multiple `identityFields` along with an
`identityTemplate` for formatting the `asset.id`
- Supports optional `identityFields` using `{ "field": "path-to-field",
"optional": true }` definition instead of a `string`.
- Supports defining key `metrics` with equations which are compatible
with the SLO product
- Supports adding `metadata` fields which will include multiple values.
- Supports `metadata` fields can be re-mapped to a new destination path
using `{ "source": "path-to-source-field", "limit": 1000, "destination":
"path-to-destination-in-output" }` definition instead of a `string`
- Supports adding `staticFields` which can also use template variables
- Support fine grain control over the frequency and sync settings for
the underlying transform
- Installs the index template components and index template settings for
the destination index
- Allow the user to configure the index patterns and timestamp field
along with the lookback
- The documents for each definition will be stored in their own index
(`.entities-observability.summary-v1.{defintion.id}`)
### Notes
- We are currently considering adding a historical index which will
track changes to the assets over time. If we choose to do this, the
summary index would remain the same but we'd add a second transform with
a group_by on the `definition.timestampField` and break the indices into
monthly indexes (configurable in the settings).
- We are looking into ways to add `firstSeenTimestamp`, this is a
difficult due to scaling issue. Essentially, we would need to find the
`minimum` timestamp for each entity which could be extremely costly on a
large datasets.
- There is nothing stopping you from creating an asset definition that
uses the `.entities-observability.summary-v1.*` index pattern to create
summaries of summaries... it can be very "meta".
### API
- `POST /api/entities/definition` - Creates a new asset definition and
starts the indexing. See examples below.
- `DELETE /api/entities/definition/{id}` - Deletes the asset definition
along with cleaning up the transform, ingest pipeline, and deletes the
destination index.
- `POST /api/entities/definition/{id}/_reset` - Resets the transform,
ingest pipeline, and destination index. This is useful for upgrading
asset definitions to new features.
## Example Definitions and Output
Here is a definition for creating services for each of the custom log
sources in the `fake_stack` dataset from `x-pack/packages/data-forge`.
```JSON
POST kbn:/api/entities/definition
{
"id": "admin-console-logs-service",
"name": "Services for Admin Console",
"type": "service",
"indexPatterns": ["kbn-data-forge-fake_stack.*"],
"timestampField": "@timestamp",
"lookback": "5m",
"identityFields": ["log.logger"],
"identityTemplate": "{{log.logger}}",
"metadata": [
"tags",
"host.name"
],
"metrics": [
{
"name": "logRate",
"equation": "A / 5",
"metrics": [
{
"name": "A",
"aggregation": "doc_count",
"filter": "log.level: *"
}
]
},
{
"name": "errorRate",
"equation": "A / 5",
"metrics": [
{
"name": "A",
"aggregation": "doc_count",
"filter": "log.level: \"ERROR\""
}
]
}
]
}
```
Which produces:
```JSON
{
"host": {
"name": [
"admin-console.prod.020",
"admin-console.prod.010",
"admin-console.prod.011",
"admin-console.prod.001",
"admin-console.prod.012",
"admin-console.prod.002",
"admin-console.prod.013",
"admin-console.prod.003",
"admin-console.prod.014",
"admin-console.prod.004",
"admin-console.prod.015",
"admin-console.prod.016",
"admin-console.prod.005",
"admin-console.prod.017",
"admin-console.prod.006",
"admin-console.prod.018",
"admin-console.prod.007",
"admin-console.prod.019",
"admin-console.prod.008",
"admin-console.prod.009"
]
},
"entity": {
"latestTimestamp": "2024-05-10T22:04:51.481Z",
"metric": {
"logRate": 37.4,
"errorRate": 1
},
"identity": {
"log": {
"logger": "admin-console"
}
},
"id": "admin-console",
"indexPatterns": [
"kbn-data-forge-fake_stack.*"
],
"definitionId": "admin-console-logs-service"
},
"event": {
"ingested": "2024-05-10T22:05:51.955691Z"
},
"tags": [
"infra:admin-console"
]
}
```
Here is an example of a definition for APM Services:
```JSON
POST kbn:/api/entities/definition
{
"id": "apm-services",
"name": "Services for APM",
"type": "service",
"indexPatterns": ["logs-*", "metrics-*"],
"timestampField": "@timestamp",
"lookback": "5m",
"identityFields": ["service.name", "service.environment"],
"identityTemplate": "{{service.name}}:{{service.environment}}",
"metadata": [
"tags",
"host.name"
],
"metrics": [
{
"name": "latency",
"equation": "A",
"metrics": [
{
"name": "A",
"aggregation": "avg",
"field": "transaction.duration.histogram"
}
]
},
{
"name": "throughput",
"equation": "A / 5",
"metrics": [
{
"name": "A",
"aggregation": "doc_count"
}
]
},
{
"name": "failedTransRate",
"equation": "A / B",
"metrics": [
{
"name": "A",
"aggregation": "doc_count",
"filter": "event.outcome: \"failure\""
},
{
"name": "B",
"aggregation": "doc_count",
"filter": "event.outcome: *"
}
]
}
]
}
```
Which produces:
```JSON
{
"host": {
"name": [
"simianhacker's-macbook-pro"
]
},
"entity": {
"latestTimestamp": "2024-05-10T21:38:22.513Z",
"metric": {
"latency": 615276.8812785388,
"throughput": 50.6,
"failedTransRate": 0.0091324200913242
},
"identity": {
"service": {
"environment": "development",
"name": "admin-console"
}
},
"id": "admin-console:development",
"indexPatterns": [
"logs-*",
"metrics-*"
],
"definitionId": "apm-services"
},
"event": {
"ingested": "2024-05-10T21:39:33.636225Z"
},
"tags": [
"_geoip_database_unavailable_GeoLite2-City.mmdb"
]
}
```
### Getting Started
The easiest way to get started is to use the`kbn-data-forge` config
below. Save this YAML to `~/Desktop/fake_stack.yaml` then run `node
x-pack/scripts/data_forge.js --config ~/Desktop/fake_stack.yaml`. Then
create a definition using the first example above.
```YAML
---
elasticsearch:
installKibanaUser: false
kibana:
installAssets: true
host: "http://localhost:5601/kibana"
indexing:
dataset: "fake_stack"
eventsPerCycle: 50
reduceWeekendTrafficBy: 0.5
schedule:
# Start with good events
- template: "good"
start: "now-1d"
end: "now-20m"
eventsPerCycle: 50
randomness: 0.8
- template: "bad"
start: "now-20m"
end: "now-10m"
eventsPerCycle: 50
randomness: 0.8
- template: "good"
start: "now-10m"
end: false
eventsPerCycle: 50
randomness: 0.8
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
- Changed the `RESULTS_API_ROUTE` to `RESULTS_INDICES_LATEST_ROUTE` with
path parameter `{pattern}`.
- Updated `getStorageResults` function to use the new route.
- Modified tests to reflect the new route and parameter usage.
- Updated server route validation to use path parameters instead of
query parameters.
closes#182868
**This is an internal route api change, so no breaking changes**
**Before**
**After:**
Resolves#178998
## Summary
This PR
- Persists group by information and apply it in the alert details page
- Adds source and tags to the alert summary field
- Fixes annotation issue on the chart by adding a margin-top
**Note**
I showed the chart title temporarily in the screenshots below for
verification: (You can do the same by removing hideTitle)
| State | Screenshot |
|---|---|
|Before|<img
src="46d39e04-f871-476f-b06b-66e7eb77db5d"
width=700 /><img
src="243ea2aa-8542-4a1f-91ff-d47c01b8452b"
width=700 />|
|After|<img
src="3b632d4f-690f-4a2f-90d6-3b9ec3d14e39"
width=700 />|
### How to test
- Create a metric threshold rule
- make sure to enable the related feature flag
```
xpack.observability.unsafe.alertDetails.observability.enabled: true
```
- Go to the alert details page and verify the charts show data related
to the selected group
- either remove hideTitle
- or make sure the data in the chart matches expectations for that
specific group
- or check the `metrics_explorer`
<img
src="20996859-0e17-44fa-a294-0c124daf849e"
width=500 />
<img
src="21beda58-0aff-42c2-a74e-df422eda347c"
width=500 />
- Create an APM Latency threshold rule and check the active alert
annotation to have the right color.
---------
Co-authored-by: Faisal Kanout <faisal@kanout.com>
Co-authored-by: Cauê Marcondes <55978943+cauemarcondes@users.noreply.github.com>
