## Summary
Fixes#212917
The root problem is belongs into the annotation layer logic to produce
the reference id for the persisted saved object.
In the previous logic a new `uuid` was generated all the time leading to
a continuous flow of `setState` calls to update the "runtime" state of
the Lens object when inline editing: the fix was to produce a stable id
in the `extractReferences` logic to avoid the re-renders.
The logic has been tweaked a bit now with some extra explanations inline
to make it more understandable.
New tests have been added to smoke test this scenario.
### Checklist
Check the PR satisfies following conditions.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Nick Partridge <nick.ryan.partridge@gmail.com>
EmbeddableInput type is part of the legacy embeddable system. The legacy
embeddable system is being removed and as such, the EmbeddableInput type
is being removed.
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR introduces validation rules for Case Observables, shared between
client and the server.
### Testing
- Create a case
- Add on observable to it, picking up the ipv4 as an observable type
(for instance)
- Verify that only the valid values are allowed.
- Try updating the observable after it is created, same validation rules
apply.
- Do the same thing using API routes.
Follow-up to: https://github.com/elastic/kibana/pull/212077
This PR includes an API test that covers `get_alerts_dataset_info` and
would have caught the bug fixed in
https://github.com/elastic/kibana/pull/212077.
It also contains the following bug fixes:
- Fix system message in `select_relevant_fields`
- Change prompt in `select_relevant_fields` so that the LLM consistently
uses the right format when responding.
## Summary
This is intended to be a "minimal" migration for Defend Insights to
langgraph + output chunking. Other than the increased events due to the
context increase from output chunking, the functionality is unchanged.
* migrates defend insights to langgraph
* adds output chunking / refinement
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
This PR makes the AI suggestions button more stable in case of
misconfigured fields:
* Only make the button clickable if there are sample values
* Filter out sample documents that don't have the required field on the
server (would have broken the request before)
<img width="344" alt="Screenshot 2025-03-04 at 15 43 23"
src="https://github.com/user-attachments/assets/12045985-cfac-4a13-a23c-595ac6503c1a"
/>
Embeddable ViewMode is part of legacy embeddable architecture. This PR
removes Embeddable ViewMode and replaces its usage with
presentation-publishing ViewMode. presentation-publishing ViewMode is a
string literal type so an enum is no longer needed.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
This PR fixes the Unstructured syslog flow. It picks up 5 samples send
them to LLM to create a pattern and tests all the samples against the
pattern , collects the unparsed samples [ if any ] , send them in for
next round of pattern check and so on.
This creates a list of patterns that matches all the samples and creates
a grok processor with those patterns and it breaks the syslogs down into
a JSON for ECS mapping , categorization and related graphs.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Summary
Fix https://github.com/elastic/kibana/issues/211514
It seems the failure is because of the order difference in
`agent_logs_top_errors`.
The other differences shouldn't matter because we are using `
expect.objectContaining` on the result object.
Enabling `consistent-type-imports` rule and running `node scripts/eslint
--no-cache --fix` to fix every type import.
Every other ML owned plugin has this rule enabled. Originally added in
https://github.com/elastic/kibana/pull/176921
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
BUG: https://github.com/elastic/security-team/issues/11783
This PR fixes the behaviour of the
`/api/security_ai_assistant/chat/complete` route where the
`conversationId` can be passed as an empty string. This may lead to
unexpected results described in
https://github.com/elastic/security-team/issues/11783#issuecomment-2696529040.
### Expected behaviour
We should throw a bad request (400) http error when empty
`conversationId` has been passed.
### Testing
* Use this `curl` command to test the endpoint.
```
curl --location 'http://localhost:5601/api/security_ai_assistant/chat/complete' \
--header 'kbn-xsrf: true' \
--header 'Content-Type: application/json' \
--data '{
"connectorId": "{{my-gpt4o-ai}}",
"conversationId": "",
"isStream": false,
"messages": [
{
"content": "Follow up",
"role": "user"
}
],
"persist": true
}'
```
You should see next error as a response:
```
{
"statusCode": 400,
"error": "Bad Request",
"message": "[request body]: conversationId: String must contain at least 1 character(s), conversationId: No empty strings allowed"
}
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Fix for
[https://github.com/elastic/kibana/security/code-scanning/546](https://github.com/elastic/kibana/security/code-scanning/546)
To fix the problem, we need to ensure that backslashes are also escaped
in the `trim_key` and `trim_value` properties of the `kvInput` object.
This can be done by adding an additional replace call to escape
backslashes before escaping single and double quotes. The best way to
fix this without changing existing functionality is to use a regular
expression with the `g` flag to replace all occurrences of backslashes
with double backslashes.
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
## Summary
The else is being called atm only in Discover ES|QL mode, when the user
is typing stats or keep.
The `isTextBased` flag is missing which means that we don't call the
`mapToOriginalColumnsTextBased` but the one for the dsl mode which might
cause bugs.
I don't think that Peter had any reason to do so, it seems more as it
slipped.
## Summary
Small follow up of https://github.com/elastic/kibana/pull/208091
The editor autocompletion added an incorrect import and so I'm removing
it, plus a few comments that should have been removed.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## 📓 Summary
When the condition is not met, the processing simulation reports wrong
metrics and fails on a unhandler error.
This work fix the issue and also update the document simulation metrics,
reporting how many documents are skipped by a processor during the
simulation.
A follow-up work will update the filters on the date to better reflect
the available states of the documents (parsed, partially parsed,
skipped, failed).
<img width="701" alt="Screenshot 2025-02-28 at 12 47 10"
src="https://github.com/user-attachments/assets/1b6979e4-78a1-4db3-af72-faaf06c0e249"
/>
## Summary
Resolves#206488. This PR introduces a new internal API that allows an
agent policy and its package policies to be created in one request:
<details>
<summary>⤵️ Expand for console snippet ⤵️</summary>
```
POST kbn:/internal/fleet/agent_and_package_policies
{
"id": "jens-awesome-policy",
"name": "awesome policy",
"description": "",
"namespace": "default",
"package_policies": [
{
"policy_ids": [
"jens-awesome-policy"
],
"package": {
"name": "log",
"version": "1.1.2"
},
"name": "log-for-awesome-policy",
"description": "",
"namespace": "",
"inputs": {
"logs-logfile": {
"enabled": true,
"streams": {
"log.log": {
"enabled": true,
"vars": {
"paths": [
"/tmp/some-path"
],
"data_stream.dataset": "generic",
"tags": [],
"custom": ""
}
}
}
}
}
},
{
"id": "fixed-id-for-filestream",
"package": {
"name": "filestream",
"version": "1.0.1"
},
"name": "filestream-1",
"description": "",
"namespace": "",
"inputs": {
"filestream-filestream": {
"enabled": true,
"streams": {
"filestream.generic": {
"enabled": true,
"vars": {
"paths": [
"/var/log/*.log"
],
"data_stream.dataset": "filestream.generic",
"parsers": "#- ndjson:\n# target: \"\"\n# message_key: msg\n#- multiline:\n# type: count\n# count_lines: 3\n",
"exclude_files": [
"\\.gz$"
],
"include_files": [],
"tags": [],
"recursive_glob": true,
"clean_inactive": -1,
"harvester_limit": 0,
"fingerprint": true,
"fingerprint_offset": 0,
"fingerprint_length": 1024,
"exclude_lines": [],
"include_lines": []
}
}
}
}
}
}
]
}
```
</details>
If successful, the response will be the agent policy with the full
nested package policies.
`id`s can be specified or omitted for either the agent or package
policy. If necessary, the resulting package policy's `policy_id` /
`policy_ids` will be updated with final ID of the created agent policy.
If any of the package policies fail to be created for any reason, there
is a rollback mechanism to delete the agent and package policies that
were already created.
The API also supports any query params that are supported by the create
agent policy and create package policy endpoints:
```
sys_monitoring: boolean; // passed to agent policy creation
format: 'simplified' | 'legacy; // passed to package policy creation
```
### Dev note
The new `createAgentAndPackagePoliciesHandler()` is unique in that it
acts mostly as a passthrough to other handlers,
`createAgentPolicyHandler` and `createPackagePolicyHandler`. This means
all the checks (spaces, rbac, etc) performed on the other handlers are
triggered appropriately.
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
<img width="600" alt="Screenshot 2025-02-25 at 11 35 32 AM"
src="https://github.com/user-attachments/assets/2567e20c-1223-4001-9866-edbe8c534350"
/>
PR does the following
* Updates maplibre to 5.1.1. Maplibre 5.0 introduced the globe
projection
* Updates sample data maps to use new globe projection
* Existing maps will use existing "mercator" projection
* Adds globe projection to newly created maps
* Adds `projection` setting, allowing users to switch between classic
"mercator" and new "globeInterpolate" projection
<img width="400" alt="Screenshot 2025-02-25 at 11 35 41 AM"
src="https://github.com/user-attachments/assets/5c55003b-dbe9-4adc-835c-0a2911c81074"
/>
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Fixes logic on fresh cluster where the ELSER was not started yet, in
this case API reports `status` as `uninstalled`, but it doesn't mean
that the Product documentation was actually uninstall, but rather it's a
default state.
Added internal `product_documentation_status` to KB status API to make
sure we keep track of the status internally and present the banner only
if the docs were intentionally uninstalled
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Relates https://github.com/elastic/kibana/issues/206242
Implemented installing integrations from the doc in the follower index.
Can be tested locally by creating the ccr index locally or setting up 2
local clusters and set up ccr between them.
To test:
- Requires https://github.com/elastic/elasticsearch/pull/123493,
checkout and run es from source
```
yarn es source --license trial -E xpack.security.authc.api_key.enabled=true -E xpack.security.authc.token.enabled=true --source-path=/Users/juliabardi/elasticsearch -E path.data=/tmp/es-data -E xpack.ml.enabled=false
```
- Enable feature flag `xpack.fleet.enableExperimental:
['enableSyncIntegrationsOnRemote']`
- Create doc in ccr index, `hosts` should match local elasticsearch host
```
POST fleet-synced-integrations-ccr-remote1/_doc
{
"id": "fleet-synced-integrations",
"remote_es_hosts": [
{
"hosts": [
"http://192.168.64.1:9200"
],
"name": "remote1",
"sync_integrations": true
}
],
"integrations": [
{
"package_version": "1.25.0",
"package_name": "nginx",
"updated_at": "2025-02-24T09:03:51.936Z"
}
]
}
```
- Wait 1m until the task runs, verify that the integrations from the doc
are installed
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
Remove extraneous dependencies:
* `canvas` was depending on 'webpack' purely for a type (dev-time).
* `@kbn/optimizer-webpack-helpers` (canvas depends on it 🤨) was
depending on 'webpack' solely for a function that could be defined in
`@kbn/optimizer` (devOnly).
Part of https://github.com/elastic/kibana/issues/200725
To track all relevant data:
> * User-facing deployment params, i.e. optimized for, VCPU level usage,
etc.
> * Resulting API params, e.g. number of allocations, etc.
This PR introduces
[EBT](https://docs.elastic.dev/telemetry/collection/event-based-telemetry)
event tracking, as ML is not currently using EBT events for data
collection, it includes the creation of `Telemetry Service`, accessible
via the `useMlKibana` hook.
I have already triggered some events for staging, which are visible
[here](https://telemetry-v2-staging.elastic.dev/s/ml/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:now-4M,to:now))&_a=(columns:!(timestamp),dataSource:(dataViewId:'74c6b7f0-3037-423f-a531-c736c70813b0',type:dataView),filters:!(),interval:auto,query:(language:kuery,query:'event_type:%20%22Trained%20Models%20Deployment%20Created%22%20'),sort:!(!(timestamp,desc)),viewMode:documents)).
Additionaly, I've created a test
[dashboard](b61d06af-4b3d-4550-8e5e-da6464652371/page/p_rssl6vh8ld/edit)
in Looker Studio to visualize the collected data.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
The esql types are around many packages and plugins creating problems
either with the bundle size or cyclic dependencies.
This PR is moving some of them on the esql-types package. I am going to
move more there in follow up PRs.
(I decided to rename it as it makes more sense to have a generic one for
esql rather one only for the variables)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Marco Vettorello <vettorello.marco@gmail.com>
## Summary
Show error to the user when trying to setup Knowledge base on undersized
cluster
<img width="1847" alt="Zrzut ekranu 2025-02-26 o 19 03 43"
src="https://github.com/user-attachments/assets/a42d8560-aebb-410e-a364-7a27074f62fc"
/>
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Garrett Spong <spong@users.noreply.github.com>
Co-authored-by: Garrett Spong <garrett.spong@elastic.co>
Fixes https://github.com/elastic/kibana/issues/211911
The CSV processing is now a three-stage process:
1. Parse the samples with the temporary column names of the form
`column1`.
2. Test parsing with the actual pipeline that parses into
`package.dataStream.columnName`.
3. Convert the samples into JSON form `{"columnName": "value", ...}` for
further processing.
Now the pipeline works as expected:
```yaml
- csv:
tag: parse_csv
field: message
target_fields:
- ai_202502211453.logs._timestamp
- ai_202502211453.logs.message
description: Parse CSV input
- drop:
ignore_failure: true
if: >-
ctx.ai_202502211453?.logs?._timestamp == '@timestamp' &&
ctx.ai_202502211453?.logs?.message == 'message'
tag: remove_csv_header
description: Remove the CSV header line by comparing the values
```
There are unit tests tests for the CSV functionality that include a mock
CSV processing pipeline.
## Summary
This PR fixes a couple of things with regards to the parsing of the
OpenAPI spec for use in CEL generation:
1) fixes and greatly simplifies the parsing of the OpenAPI spec so that
we collect all the $ref tags in the response object
2) only collects the top level schemas from the response object (since
that's all we really need for the CEL program)
3) fixes it so that users cannot select 'Save configuration' if there is
a generation error
4) better error messaging if/when a spec parsing error occurs
Note re fix # 3, the 'Save configuration' button will still initially be
available upon an error occurring. Then when if the user tries to click
save after an error, it will then disable the save button and show the
message indicating they need a successful generation to save. This is
consistent with the UX philosophy in the rest of the flyout that all
buttons are enabled by default, and if the user does something 'wrong',
we then provide guidance for how to proceed.
Relates: https://github.com/elastic/kibana/issues/210271
## Screenshots
<details>
<summary>parsing fix</summary>
<img width="450" alt="Screenshot 2025-02-21 at 2 15 34 PM"
src="https://github.com/user-attachments/assets/80fe8e56-ffe3-4d5c-b6ac-5a57e025b70b"
/>
</details>
<details>
<summary>save disabled fix</summary>
<img width="450" alt="Screenshot 2025-02-21 at 2 13 45 PM"
src="https://github.com/user-attachments/assets/5220bad7-70b1-4ade-83f7-ce1f97d115d1"
/>
<img width="450" alt="Screenshot 2025-02-21 at 2 13 55 PM"
src="https://github.com/user-attachments/assets/427bb52c-6fa9-457f-ab28-f490be981094"
/>
</details>
