This PR resolves https://github.com/elastic/search-team/issues/7993
Recently, a backend https://github.com/elastic/elasticsearch/pull/111366
was merged to elasticsearch which basically replaces 'model_id' with
'inference_id' in the GET inference api. This change was also backported
to v8.15. Due to this change, the frontend works related to
semantic_text UI and Inference Endpoints view are broken.
---------
Co-authored-by: Dima Arnautov <dmitrii.arnautov@elastic.co>
Closes https://github.com/elastic/kibana/issues/187348
This changes adds an optional `history.settings.lookbackPeriod` property
that will default to `1h` if none is provided. The main point is to
prevent accidental processing of the entire dataset when creating a
definition.
I took the opportunity to do some refactoring:
- `durationSchema` was transforming a literal duration (eg `1h`) into a
`moment.Duration` with overriden `toJSON` property. since we don't use
any of the `moment` functionalities in consuming code the schema now
returns the raw string after regex validation
- split the `generateHistoryTransform` in `generateHistoryTransform` and
`generateBackfillHistoryTransform`
**Resolves:** https://github.com/elastic/kibana/issues/189269
**Resolves:** https://github.com/elastic/kibana/issues/189270
## Summary
This PR adds an ability to specify OpenAPI `servers` and security requirements (`security`) to be used in the result bundle. `servers` and/or `security` in the source OpenAPI specs are be dropped when custom `servers` and/or `security` provided.
## Details
Kibana is usually deployed at a single access point and manages authentication in a central way. That way it's much more convenient to have control on what `servers` and `security` are present in the result bundles. It will help to avoid conflicts, duplicates and update them in centralized way.
This PR extends OpenAPI bundler configuration options with `prototypeDocument`. "Prototype" in the name means it's a prototype for the result. The bundler uses certain properties from that prototype OpenAPI document to add them to the result OpenAPI bundle. The following properties are used
- `info` representing OpenAPI Info object (former `options.specInfo`)
- `servers` OpenAPI Server Object Array
- `security` + `components.securitySchemes` OpenAPI Security Requirement Object Array + OpenAPI Security Schemes Object (validation checks that both fields are set otherwise an error is thrown)
For convenience `prototypeDocument` could be specified as a string path to a file containing prototype OpenAPI document.
## How to test?
`prototypeDocument` can be specified for `bundle` and `merge` utilities like the following
**bundle**
```js
const { bundle } = require('@kbn/openapi-bundler');
(async () => {
await bundle({
sourceGlob: 'source/glob/*.yaml',
outputFilePath: 'output/bundle.yaml,
options: {
prototypeDocument: {
info: {
title: 'Some title',
description: 'Some description',
},
servers: [{
url: 'https://{kibana_url}',
variables: {
kibana_url: {
default: 'localhost:5601',
}
}
}],
security: [{ ApiKeyAuth: [] }],
components: {
securitySchemes: {
ApiKeyAuth: {
type: 'apiKey',
in: 'header',
name: 'Authorization',
}
}
}
},
},
});
```
**bundle** with external prototype document
```js
const { bundle } = require('@kbn/openapi-bundler');
(async () => {
await bundle({
sourceGlob: 'source/glob/*.yaml',
outputFilePath: 'output/bundle.yaml,
options: {
prototypeDocument: 'path/to/prototype_document.yaml',,
},
});
```
**merge**
```js
const { merge } = require('@kbn/openapi-bundler');
(async () => {
await merge({
sourceGlobs: [
'absolute/path/to/file.yaml`,
'some/glob/*.schema.yaml',
],
outputFilePath: 'output/file/path/bundle.yaml',
options: {
prototypeDocument: {
info: {
title: 'Some title',
description: 'Some description',
},
servers: [{
url: 'https://{kibana_url}',
variables: {
kibana_url: {
default: 'localhost:5601',
}
}
}],
security: [{ ApiKeyAuth: [] }],
components: {
securitySchemes: {
ApiKeyAuth: {
type: 'apiKey',
in: 'header',
name: 'Authorization',
}
}
}
},
},
});
})();
```
**merge** with external prototype document
```js
const { merge } = require('@kbn/openapi-bundler');
(async () => {
await merge({
sourceGlobs: [
'absolute/path/to/file.yaml`,
'some/glob/*.schema.yaml',
],
outputFilePath: 'output/file/path/bundle.yaml',
options: {
prototypeDocument: 'path/to/prototype_document.yaml',
},
});
})();
```
The result bundles will contain specified `servers` and `security` while source `servers` and `security` will be dropped.
## Summary
Part of #187684.
So far the popover to filter fields was only available when grouping was
enabled. This PR updates the behavior so it's available all the time and
can be used to exclude field candidates from the analysis. If we detect
the index to be based on an ECS schema, we auto-select a set of
predefined fields.
Changes in this PR:
- Creates a new route
`/internal/aiops/log_rate_analysis/field_candidates` to be able to fetch
field candidates independent of the main streaming API call.
- Fixes the code to consider "remaining" field candidates to also
consider text field candidates. This was originally developed to allow
to continue an analysis that errored for some reason. We use that option
to also pass on the custom field list from the field selection popover.
- Fetching the field candidates is done in a new redux slice
`logRateAnalysisFieldCandidatesSlice` using an async thunk.
- Filters the list of field candidates by a predefined field of allowed
fields when an ECS schema gets detected.
- Renames `fieldCandidates` to `keywordFieldCandidates` for clearer
distinction against `textFieldCandidates`.
- Refactors `getLogRateAnalysisTypeForCounts` args to a config object.
- Bump the API version for the full log rate analysis to version 3. We
missed bumping the version in
https://github.com/elastic/kibana/pull/188648. This update manages
proper versioning between v2 and v3, also the API integration tests
cover both versions.
[aiops-log-rate-analysis-fields-filter-0001.webm](https://github.com/user-attachments/assets/e3ed8d5b-f01c-42ef-8033-caa7135b8cc0)
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Summary
At the moment, our package generator creates all packages with the type
`shared-common`. This means that we cannot enforce boundaries between
server-side-only code and the browser, and vice-versa.
- [x] I started fixing `packages/core/*`
- [x] It took me to fixing `src/core/` type to be identified by the
`plugin` pattern (`public` and `server` directories) vs. a package
(either common, or single-scoped)
- [x] Unsurprisingly, this extended to packages importing core packages
hitting the boundaries eslint rules. And other packages importing the
latter.
- [x] Also a bunch of `common` logic that shouldn't be so _common_ 🙃
### For maintainers
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Adds Amazon Bedrock support to the [Inference Endpoints management
UI](https://github.com/elastic/kibana/pull/186206)
(`relevance/inference_endpoints`) management list view.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
**Resolves:** https://github.com/elastic/kibana/issues/188817
## Summary
This PR adds automatic shared components conflict resolution functionality for OpenAPI merger. It boils down to a similar result as `npx @redocly/cli join --prefix-components-with-info-prop title` produces by prefixing shared components with document's title in each source.
OpenAPI bundler intentionally won't solve conflicts automatically since it's focused on bundling domain APIs where conflicts are usually indicators of upstream problems.
## Details
While working with various OpenAPI specs it may happen that different specs use exactly the same name for some shared components but different definitions. It must be avoided inside one API domain but it's a usual situation when merging OpenAPI specs of different API domains. For example domains may define a shared `Id` or `404Response` schemas where `Id` is a string in one domain and a number in another.
OpenAPI merger implemented in https://github.com/elastic/kibana/pull/188110 and OpenAPI bundler implemented in https://github.com/elastic/kibana/pull/171526 do not solve shared components related conflicts automatically. It works perfectly for a single API domain forcing engineers choosing shared schema names carefully.
This PR adds automatic shared components conflict resolution for OpenAPI merger. It prefixes shared component names with a normalized document's title.
OpenAPI bundler intentionally won't solve conflicts automatically since it's focused on bundling domain APIs where conflicts are usually indicators of upstream problems.
## Example
Consider two following OpenAPI specs each defining local `MySchema`
**spec1.schema.yaml**
```yaml
openapi: 3.0.3
info:
title: My endpoint
version: '2023-10-31'
paths:
/api/some_api:
get:
operationId: MyEndpointGet
responses:
'200':
content:
application/json:
schema:
$ref: '#/components/schemas/MySchema'
components:
schemas:
MySchema:
type: string
enum:
- value1
```
**spec2.schema.yaml**
```yaml
openapi: 3.0.3
info:
title: My another endpoint
version: '2023-10-31'
paths:
/api/another_api:
get:
operationId: MyAnotherEndpointGet
responses:
'200':
content:
application/json:
schema:
$ref: '#/components/schemas/MySchema'
components:
schemas:
MySchema:
type: number
```
and a script to merge them
```js
require('../../src/setup_node_env');
const { resolve } = require('path');
const { merge } = require('@kbn/openapi-bundler');
const { REPO_ROOT } = require('@kbn/repo-info');
(async () => {
await merge({
sourceGlobs: [
`${REPO_ROOT}/oas_docs/spec1.schema.yaml`,
`${REPO_ROOT}/oas_docs/spec2.schema.yaml`,
],
outputFilePath: resolve(`${REPO_ROOT}/oas_docs/merged.yaml`),
options: {
mergedSpecInfo: {
title: 'Merge result',
version: 'my version',
},
},
});
})();
```
will be merged successfully to
**merged.yaml**
```yaml
openapi: 3.0.3
info:
title: Merge result
version: 'my version'
paths:
/api/another_api:
get:
operationId: MyAnotherEndpointGet
responses:
'200':
content:
application/json; Elastic-Api-Version=2023-10-31:
schema:
$ref: '#/components/schemas/My_another_endpoint_MySchema'
/api/some_api:
get:
operationId: MyEndpointGet
responses:
'200':
content:
application/json; Elastic-Api-Version=2023-10-31:
schema:
$ref: '#/components/schemas/My_endpoint_MySchema'
components:
schemas:
My_another_endpoint_MySchema:
type: number
My_endpoint_MySchema:
enum:
- value1
type: string
```
Related to #183220
## Summary
This PR extracts `getEcsGroups` to a package to save ECS groups in the
Alert As Data (AAD) document for the metric threshold rule.
### 🧪 How to test
- Create a metric threshold rule with multiple groups (both ECS and
non-ECS fields)
- Check the related AAD document; you should be able to see the ECS
fields at the root level and not see non-ECS fields there
- Check the same information for the recovered alerts
- Rules without group by should work as before
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Follow up after:
- https://github.com/elastic/kibana/pull/188509
Contributes to:
- https://github.com/elastic/security-team/issues/9954
This PR adds the tooltip to the Distribution Bar component. Tooltip
logic is the following:
- when the bar section is hovered, tooltip is shown
- by default the most right tooltip is shown, when another section is
hovered, the most right tooltip is hidden
- numbers on the tooltip display as shortened when >1000
- label can be a React Node to support passing translation components
Not covered in the PR:
- edge case when the tooltip goes out of the left boundary of the whole
component, it should change direction
### Screenshot
<img width="1128" alt="Screenshot 2024-07-19 at 14 42 36"
src="https://github.com/user-attachments/assets/533ad54b-c931-42bb-be45-78a602ffa6d4">
### How to test
run `yarn storybook security_solution_packages` and open storybook on
`http://localhost:9001/`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Paulo Henrique <paulo.henrique@elastic.co>
**Addresses**: https://github.com/elastic/kibana/issues/184428
## Summary
This PR adds scripts for automatic bundling of AI Assistant API OpenAPI specs as a part of PR pipeline. Corresponding result bundles are automatically committed to the Assistant Common package `x-pack/packages/kbn-elastic-assistant-common` in the `docs/openapi/ess/` and `docs/openapi/serverless` folders (similar to https://github.com/elastic/kibana/pull/186384).
## Summary
This PR adds 2 new optional settings for the history section of the
entity definition to enable a backfill transform:
- `history.settings.backfillSyncDelay` – A duration format, i.e. `15m`,
that enables the backfill transform and sets the sync delay to whatever
duration the user has configured
- `history.settings.backfilLookbackPeriod` – Controls how far back the
transform will start processing documents.
The idea behind this transform is that it will run with a longer delay
than the default transform. If there are events that show up after the
default transform's checkpoint has moved on, the backfill transform will
make a second sweep to backfill any data the default transform had
missed.
### Testing
Save the following config to `fake_logs.delayed.yaml`
```YAML
---
elasticsearch:
installKibanaUser: false
kibana:
installAssets: true
host: "http://localhost:5601/kibana"
indexing:
dataset: "fake_logs"
eventsPerCycle: 100
artificialIndexDelay: 300000
schedule:
- template: "good"
start: "now-1d"
end: false
eventsPerCycle: 100
```
run `node x-pack/scripts/data_forge.js --config fake_logs.delayed.yaml`
then run the following in Kibana's "Dev Tools":
```JSON
POST kbn:/internal/api/entities/definition
{
"id": "fake-logs-services-no-backfill",
"name": "Services for Fake Logs",
"type": "service",
"version": "0.0.1",
"indexPatterns": ["kbn-data-forge-fake_logs.*"],
"history": {
"timestampField": "@timestamp",
"interval": "1m"
},
"identityFields": ["labels.groupId", "labels.eventId"],
"displayNameTemplate": "{{labels.groupId}}:{{labels.eventId}}",
"metadata": [
"host.name"
],
"metrics": [
{
"name": "latency",
"equation": "A",
"metrics": [
{
"name": "A",
"aggregation": "avg",
"field": "event.duration"
}
]
},
{
"name": "logRate",
"equation": "A",
"metrics": [
{
"name": "A",
"aggregation": "doc_count",
"filter": "log.level: *"
}
]
},
{
"name": "errorRate",
"equation": "A",
"metrics": [
{
"name": "A",
"aggregation": "doc_count",
"filter": "log.level: \"error\""
}
]
}
]
}
POST kbn:/internal/api/entities/definition
{
"id": "fake-logs-services-with-backfill",
"name": "Services for Fake Logs",
"type": "service",
"version": "0.0.1",
"indexPatterns": ["kbn-data-forge-fake_logs.*"],
"history": {
"timestampField": "@timestamp",
"interval": "1m",
"settings": {
"backfillSyncDelay": "10m",
"backfillLookback": "24h"
}
},
"identityFields": ["labels.groupId", "labels.eventId"],
"displayNameTemplate": "{{labels.groupId}}:{{labels.eventId}}",
"metadata": [
"host.name"
],
"metrics": [
{
"name": "latency",
"equation": "A",
"metrics": [
{
"name": "A",
"aggregation": "avg",
"field": "event.duration"
}
]
},
{
"name": "logRate",
"equation": "A",
"metrics": [
{
"name": "A",
"aggregation": "doc_count",
"filter": "log.level: *"
}
]
},
{
"name": "errorRate",
"equation": "A",
"metrics": [
{
"name": "A",
"aggregation": "doc_count",
"filter": "log.level: \"error\""
}
]
}
]
}
```
The first transform should end up giving you history every 5 minutes,
the second will backfill and give you history every minute up until ~10
minutes. If you where to create a dashboard with the document counts for
the last hour, it would look like this:
## Summary
Adopted `BedrockChat` from `@langchain/community` package that adds
support for tools calling
https://js.langchain.com/v0.2/docs/integrations/chat/bedrock/
Adopted `ChatGoogleGenerativeAI ` from `@langchain/google-genai` package
that adds support for tools calling
https://js.langchain.com/v0.2/docs/integrations/chat/google_generativeai
Hidden behind FF:
`--xpack.securitySolution.enableExperimental=[assistantBedrockChat]`
As of this PR `integration_assistant` is still going to use
`ActionsClientSimpleChatModel`. After the FF will be enabled by default
we will switch `integration_assistant` to use new chat model.
Thank you @stephmilovic a ton 🙇
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Steph Milovic <stephanie.milovic@elastic.co>
Co-authored-by: Garrett Spong <spong@users.noreply.github.com>
This PR adds the `service.logs` dataset to Data Forge . The EEM project
needs this dataset to test the default service logs entity definition.
This dataset is different because I wanted to create a fully compliant
data stream. This change also includes changes to the name of the
default ingest pipeline to `logs@custom` to work with the `logs-*-*`
component templates and ingest pipelines. If a document has
`data_stream.dataset` it will now be routed to
`logs-${doc.data_stream.dataset}-default`. If the document has
`data_stream.type`, `data_stream.dataset`, and `data_stream.namespace`
it will be index to `{type}-{dataset}-{namespace}` following the default
data stream conventions.
Because I've changed the name of the ingest pipeline, I also had to
update the index templates for the other datasets.
### Testing
Use the following YAML:
```yaml
---
elasticsearch:
installKibanaUser: false
kibana:
installAssets: false
host: "http://localhost:5601/kibana"
indexing:
dataset: "service.logs"
eventsPerCycle: 100
interval: 10000
schedule:
- template: "good"
start: "now-1h"
end: false
eventsPerCycle: 100
```
Click on "Logs" under "Observability", you should see something like:
<img width="2048" alt="image"
src="https://github.com/user-attachments/assets/64837c5c-9380-4897-9ccc-acae313683ee">
To check the other data source, change `dataset` to `fake_stack`,
`fake_logs`, `fake_hosts` and check `event.ingested` is set on the
documents; none of these show up in the "Logs Explorer", you'll have to
check them out seperately via "Dev Console".
## Summary
Part of #187684.
This moves functions related to log rate changes to the
`@kbn/aiops_log_rate_analysis` package.
- `getLogRateAnalysisType` was renamed to
`getLogRateAnalysisTypeForHistogram` to indicate its use with histogram
data.
- `getLogRateAnalysisTypeForCounts` was added for cases where we don't
have the histogram data available but just the doc counts for baseline
an deviation time ranges. This isn't used yet as of this PR but will be
in a follow up in combination with the o11y AI assistant.
- `getSwappedWindowParameters` is a helper to consolidate inline code
that's used to swap baseline and deviation when we detected a dip in log
rate.
- Rounding for the log rate change messages was tweaked. Changes below
`10x` will now be rounded to one digit to avoid messages like `1x
increase`.
- Tweaked/Shortened the message for 0 in baseline or deviation to just
`45 up from 0 in baseline` / `down to 0 from 45 in baseline`.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Summary
This PR closes https://github.com/elastic/kibana/issues/188171 by
converting the route validate to Zod for `get`, `reset`, and `delete`
APIs. This also changes the validation for the `create` API to use
`buildRouteValidationWithZod` along with adding `strict()` to each of
the schemas.
Closes https://github.com/elastic/elastic-entity-model/issues/103
---------
Co-authored-by: Kevin Lacabane <kevin.lacabane@elastic.co>
Runtime mappings need to be passed to the categorization request factory
function and the field validation function.
Initially they were excluded because we only allow pattern analysis on
text fields and it is not possible to create a text runtime field.
However it is possible to apply a filter which uses a runtime field and
doing so causes pattern analysis to fail.
@walterra I have not investigated log rate analysis' behaviour, in this
PR I have just updated the call to `createCategoryRequest` to pass
`undefined`
To test, create a runtime mapping in the data view. Use this in the
query bar or in a filter in Discover and ML's Log Pattern Analysis page.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Contributes to:
- https://github.com/elastic/security-team/issues/9954
The PR contains the base for the `DistributionBar` component to be used
in the new Entity Flyout Insights.
Not included:
- badges per distribution with the number of documents and pretty names
- on hover interaction
## Screenshots
<img width="980" alt="Screenshot 2024-07-17 at 15 13 48"
src="https://github.com/user-attachments/assets/f2ca53ee-c054-4923-aa3f-7dd4017754cb">
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This PR is a follow up to #187901 – It restores the `await
queue.drain()` function call in the `createEvents()` method. Without the
`queue.drain()` call, the script will run out of memory when indexing
more than 24 hours of data because it will generate events faster than
the queue can drain them and eventually run out of memory.
This PR closes
https://github.com/elastic/elastic-entity-model/issues/116 by ensuring
that `destination` is always set when the schema is parsed along with
ensuring that if for some reason desitnation is not set, we fallback in
the actual metadata code as well. I also added a unit test for each of
the different `metadata` formats:
- String
- Object with only `source`
- Object with `source` and `limit`
- Object with `source`, `limit`, and `destination`
---------
Co-authored-by: Chris Cowan <chris@elastic.co>
Co-authored-by: Chris Cowan <chris@chriscowan.us>
Co-authored-by: Nathan L Smith <nathan.smith@elastic.co>
## Summary
This PR enables the automatic setup of the Knowledge Base and LangGraph
code paths for the `8.15` release. These features were behind the
`assistantKnowledgeBaseByDefault` feature flag, which will remain as a
gate for upcoming Knowledge Base features that were not ready for this
release.
As part of these changes, we now only support the new LangGraph code
path, and so were able to clean up the non-kb and non-RAGonAlerts code
paths. All paths within the `post_actions_executor` route funnel to the
LangGraph implementation.
> [!NOTE]
> We were planning to do the switch to the new
[`chat/completions`](https://github.com/elastic/kibana/pull/184485/files)
public API, however this would've required additional refactoring since
the API's slightly differ. We will make this change and delete the
`post_actions_executor` route for the next release.
### Checklist
Delete any items that are not applicable to this PR.
- [X] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- Working with docs team to ensure updates before merging, cc
@benironside
- [X] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Steph Milovic <stephanie.milovic@elastic.co>
## Summary
This change includes updates to the builtin service definition:
- removal of the high cardinality metadata fields until we have a
solution in place
- fetching of the metrics from the preaggregated apm metrics
- fixed metrics aggregations
- increased history transform frequency and delay to cover for delayed
ingestion
---------
Co-authored-by: Milton Hultgren <milton.hultgren@elastic.co>
Co-authored-by: Søren Louv-Jansen <sorenlouv@gmail.com>
## Summary
This PR adds a new setting, `indexing.artificialIndexDelay`, to the
indexing configuration to control how much artificial delay to add to
the timestamps. This PR also adds a "final" ingest pipeline to each data
source along with injecting a new base `component_template` which
includes the `event.ingested` field.
The artificial delay is useful for testing transforms on data that has a
significant delays. It also allows us to test if we miss data when
syncing on the transforms using `event.ingested`.
- Installs default ingest pipeline to add event.ingested to each
document
- Adds final_pipeline to each install_index_template
- Inject base component_template to each index_template at install time
- Add artificial delay for "current" events, historical events are
ingested without delay.
- Change index math to produce monthly indices
### How to test:
Copy the following to `fake_logs.delayed.yaml`:
```YAML
---
elasticsearch:
installKibanaUser: false
kibana:
installAssets: true
host: "http://localhost:5601/kibana"
indexing:
dataset: "fake_logs"
eventsPerCycle: 100
artificialIndexDelay: 300000
schedule:
- template: "good"
start: "now-1h"
end: false
eventsPerCycle: 100
```
Then run `node x-pack/scripts/data_forge.js --config
fake_logs.delayed.yaml`. This should index an hour of data immediately,
then add a 300s delay when indexing in "real time". The logs will look
like:
```
info Starting index to http://localhost:9200 with a payload size of 10000 using 5 workers to index 100 events per cycle
info Installing index templates (fake_logs)
info Installing components for fake_logs (fake_logs_8.0.0_base,fake_logs_8.0.0_event,fake_logs_8.0.0_log,fake_logs_8.0.0_host,fake_logs_8.0.0_metricset)
info Installing index template (fake_logs)
info Indexing "good" events from 2024-07-09T16:23:36.803Z to indefinitely
info Delaying 100 by 300000ms
info Waiting 60000ms
info { took: 2418721239, latency: 541, indexed: 6000 } Indexing 6000 documents.
...
```
Then after `300s`, it will index another `100` documents every `60s`.
You can also inspect the delay per minute using the following ES|QL in
Discover:
```
FROM kbn-data-forge-fake_logs.fake_logs-* | eval diff=DATE_DIFF("seconds", @timestamp, event.ingested) | STATS delay=AVG(diff) by timestamp=BUCKET(@timestamp, 1 minute)
```
This should give you a chart that looks something like this:
<img width="1413" alt="image"
src="2f48cb85-a410-487e-8f3b-41311ff95186">
There should also be a 5 minute gap at the end in Discover:
<img width="1413" alt="image"
src="660acc87-6958-4ce9-a544-d66d56f805dd">
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
In this PR Security Gen AI related APIs are changed from internal to
public.
Conversations APIs:
- POST/PUT/GET/DELETE
`"/api/security_ai_assistant/current_user/conversations/{id}"`
- GET `"/api/security_ai_assistant/current_user/conversations/_find"`
Prompts APIs:
- POST `"/api/security_ai_assistant/prompts/_bulk_action"`
- GET `"/api/security_ai_assistant/current_user/conversations/_find"`
Anonymization APIs:
- POST `"/api/security_ai_assistant/anonymization_fields/_bulk_action"`
- GET `"/api/security_ai_assistant/anonymization_fields/_find"`
## Summary
This makes semantic text work with non-root level reference fields. It
also correctly adds copy_to to existing copy_to fields instead of
replacing them, and streamlines a lot of the code.
To test these changes:
- Create an index
- Go to the index mappings page at
`app/management/data/index_management/indices/index_details?{yourIndexName}=blah&tab=mappings`
- Add an object field with a text field inside
- Add a semantic text field referencing that text field
- If you're on a Macbook, create a new inference endpoint with the model
`.elser_model_2` instead of using the default inference endpoint.
- Add a second semantic text field referencing that text field
- Save your mappings
- Use JSON view to verify that the newly created text field contains a
`copy_to` field referencing both newly created semantic text fields
- Verify that the newly created semantic text fields are also in the
JSON view
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [x] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [x] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [x] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
This PR consists of the following changes:
- An option to delete an existing inference endpoint
- Filtering the endpoints based on 'provider' and 'type'
- Search option
- Display the trained models deployment status
- Display additional 3rd party providers (Mistral, Azure OpenAI, Azure
AI Studio)
- Add licensing for gating enterprise licensed users
### Stack Management
### Serverless
---------
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
