b09d0fca86
# Backport This will backport the following commits from `main` to `8.16`: - [[Attack discovery] Fix error handling in LM studio (#213855)](https://github.com/elastic/kibana/pull/213855) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Patryk Kopyciński","email":"contact@patrykkopycinski.com"},"sourceCommit":{"committedDate":"2025-03-12T02:06:48Z","message":"[Attack discovery] Fix error handling in LM studio (#213855)\n\n## Summary\n\nError were not properly propagated to the user and instead of meaningful\nmessage we were displaying just `API Error`.\n\n<img width=\"1813\" alt=\"Zrzut ekranu 2025-03-11 o 03 47 59\"\nsrc=\"https://github.com/user-attachments/assets/8d059159-f020-4944-a463-b10799e7fa46\"\n/>\n \n \n \n\nSteps to reproduce, Thank you @andrew-goldstein 🙇 \n\n**Desk testing**\n\nTo reproduce:\n\n1. In LM Studio, download the `MLX` variant (optimized for Mac) of\n`Llama-3.2-3B-Instruct-4bit`:\n\n```\nmlx-community/Llama-3.2-3B-Instruct-4bit\n```\n\n2. Configure the model to have a context length of `131072` tokens, as\nillustrated by the screenshot below:\n\n\n\n\n3. Serve ONLY the model above in LM Studio. (Ensure no other models are\nrunning in LM Studio), as illustrated by the screenshot below:\n\n\n\n\n4. Configure a connector via the details in\n<https://www.elastic.co/guide/en/security/current/connect-to-byo-llm.html>\n\nbut change:\n\n```\nlocal-model\n```\n\nto the name of the model when configuring the connector:\n\n```\nllama-3.2-3b-instruct\n```\n\nas illustrated by the screenshot below:\n\n\n\n\n5. Generate Attack discoveries\n\n**Expected results**\n\n- Generation does NOT fail with the error described in the later steps\nbelow.\n- Progress on generating discoveries is visible in Langsmith, as\nillustrated by the screenshot below:\n\n\n\n\nNote: `Llama-3.2-3B-Instruct-4bit` may not reliably generate Attack\ndiscoveries, so generation may still fail after `10` generation /\nrefinement steps.\n\n6. In LM studio, serve a _second_ model, as illustrated by the\nscreenshot below:\n\n\n\n\n7. Once again, generate Attack discoveries\n\n**Expected results**\n\n- Generation does NOT fail with the errors below\n- Progress on generating discoveries is visible in Langsmith, though as\nnoted above, generation may still fail after `10` attempts if the model\ndoes not produce output that conforms to the expected schema\n\n**Actual results**\n\n- Generation fails with an error similar to:\n\n```\ngenerate node is unable to parse (openai) response from attempt 0; (this may be an incomplete response from the model): Status code: 400. Message: API Error:\nBad Request: ActionsClientLlm: action result status is error: an error occurred while running the action - Status code: 400. Message: API Error: Bad Request,\n```\n\nor\n\n```\ngenerate node is unable to parse (openai) response from attempt 0; (this may be an incomplete response from the model): Status code: 404. Message: API Error: Not Found - Model \"llama-3.2-3b-instruct\" not found. Please specify a valid model.\n```\n\nas illustrated by the following screenshot:\n\n\n","sha":"0b9cceb57413ee84c2b951a65d1c8b66523fbd87","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:skip","backport:prev-major","Team:Security Generative AI","Feature:Attack Discovery","backport:current-major","v9.1.0"],"title":"[Attack discovery] Fix error handling in LM studio","number":213855,"url":"https://github.com/elastic/kibana/pull/213855","mergeCommit":{"message":"[Attack discovery] Fix error handling in LM studio (#213855)\n\n## Summary\n\nError were not properly propagated to the user and instead of meaningful\nmessage we were displaying just `API Error`.\n\n<img width=\"1813\" alt=\"Zrzut ekranu 2025-03-11 o 03 47 59\"\nsrc=\"https://github.com/user-attachments/assets/8d059159-f020-4944-a463-b10799e7fa46\"\n/>\n \n \n \n\nSteps to reproduce, Thank you @andrew-goldstein 🙇 \n\n**Desk testing**\n\nTo reproduce:\n\n1. In LM Studio, download the `MLX` variant (optimized for Mac) of\n`Llama-3.2-3B-Instruct-4bit`:\n\n```\nmlx-community/Llama-3.2-3B-Instruct-4bit\n```\n\n2. Configure the model to have a context length of `131072` tokens, as\nillustrated by the screenshot below:\n\n\n\n\n3. Serve ONLY the model above in LM Studio. (Ensure no other models are\nrunning in LM Studio), as illustrated by the screenshot below:\n\n\n\n\n4. Configure a connector via the details in\n<https://www.elastic.co/guide/en/security/current/connect-to-byo-llm.html>\n\nbut change:\n\n```\nlocal-model\n```\n\nto the name of the model when configuring the connector:\n\n```\nllama-3.2-3b-instruct\n```\n\nas illustrated by the screenshot below:\n\n\n\n\n5. Generate Attack discoveries\n\n**Expected results**\n\n- Generation does NOT fail with the error described in the later steps\nbelow.\n- Progress on generating discoveries is visible in Langsmith, as\nillustrated by the screenshot below:\n\n\n\n\nNote: `Llama-3.2-3B-Instruct-4bit` may not reliably generate Attack\ndiscoveries, so generation may still fail after `10` generation /\nrefinement steps.\n\n6. In LM studio, serve a _second_ model, as illustrated by the\nscreenshot below:\n\n\n\n\n7. Once again, generate Attack discoveries\n\n**Expected results**\n\n- Generation does NOT fail with the errors below\n- Progress on generating discoveries is visible in Langsmith, though as\nnoted above, generation may still fail after `10` attempts if the model\ndoes not produce output that conforms to the expected schema\n\n**Actual results**\n\n- Generation fails with an error similar to:\n\n```\ngenerate node is unable to parse (openai) response from attempt 0; (this may be an incomplete response from the model): Status code: 400. Message: API Error:\nBad Request: ActionsClientLlm: action result status is error: an error occurred while running the action - Status code: 400. Message: API Error: Bad Request,\n```\n\nor\n\n```\ngenerate node is unable to parse (openai) response from attempt 0; (this may be an incomplete response from the model): Status code: 404. Message: API Error: Not Found - Model \"llama-3.2-3b-instruct\" not found. Please specify a valid model.\n```\n\nas illustrated by the following screenshot:\n\n\n","sha":"0b9cceb57413ee84c2b951a65d1c8b66523fbd87"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/213855","number":213855,"mergeCommit":{"message":"[Attack discovery] Fix error handling in LM studio (#213855)\n\n## Summary\n\nError were not properly propagated to the user and instead of meaningful\nmessage we were displaying just `API Error`.\n\n<img width=\"1813\" alt=\"Zrzut ekranu 2025-03-11 o 03 47 59\"\nsrc=\"https://github.com/user-attachments/assets/8d059159-f020-4944-a463-b10799e7fa46\"\n/>\n \n \n \n\nSteps to reproduce, Thank you @andrew-goldstein 🙇 \n\n**Desk testing**\n\nTo reproduce:\n\n1. In LM Studio, download the `MLX` variant (optimized for Mac) of\n`Llama-3.2-3B-Instruct-4bit`:\n\n```\nmlx-community/Llama-3.2-3B-Instruct-4bit\n```\n\n2. Configure the model to have a context length of `131072` tokens, as\nillustrated by the screenshot below:\n\n\n\n\n3. Serve ONLY the model above in LM Studio. (Ensure no other models are\nrunning in LM Studio), as illustrated by the screenshot below:\n\n\n\n\n4. Configure a connector via the details in\n<https://www.elastic.co/guide/en/security/current/connect-to-byo-llm.html>\n\nbut change:\n\n```\nlocal-model\n```\n\nto the name of the model when configuring the connector:\n\n```\nllama-3.2-3b-instruct\n```\n\nas illustrated by the screenshot below:\n\n\n\n\n5. Generate Attack discoveries\n\n**Expected results**\n\n- Generation does NOT fail with the error described in the later steps\nbelow.\n- Progress on generating discoveries is visible in Langsmith, as\nillustrated by the screenshot below:\n\n\n\n\nNote: `Llama-3.2-3B-Instruct-4bit` may not reliably generate Attack\ndiscoveries, so generation may still fail after `10` generation /\nrefinement steps.\n\n6. In LM studio, serve a _second_ model, as illustrated by the\nscreenshot below:\n\n\n\n\n7. Once again, generate Attack discoveries\n\n**Expected results**\n\n- Generation does NOT fail with the errors below\n- Progress on generating discoveries is visible in Langsmith, though as\nnoted above, generation may still fail after `10` attempts if the model\ndoes not produce output that conforms to the expected schema\n\n**Actual results**\n\n- Generation fails with an error similar to:\n\n```\ngenerate node is unable to parse (openai) response from attempt 0; (this may be an incomplete response from the model): Status code: 400. Message: API Error:\nBad Request: ActionsClientLlm: action result status is error: an error occurred while running the action - Status code: 400. Message: API Error: Bad Request,\n```\n\nor\n\n```\ngenerate node is unable to parse (openai) response from attempt 0; (this may be an incomplete response from the model): Status code: 404. Message: API Error: Not Found - Model \"llama-3.2-3b-instruct\" not found. Please specify a valid model.\n```\n\nas illustrated by the following screenshot:\n\n\n","sha":"0b9cceb57413ee84c2b951a65d1c8b66523fbd87"}}]}] BACKPORT--> Co-authored-by: Patryk Kopyciński <contact@patrykkopycinski.com> |
||
|---|---|---|
| .. | ||
| __mocks__/@kbn/code-editor | ||
| common | ||
| public | ||
| server | ||
| jest.config.js | ||
| kibana.jsonc | ||
| README.md | ||
| tsconfig.json | ||
Stack Connectors
The stack_connectors plugin provides connector types shipped with Kibana, built on top of the framework provided in the actions plugin.
Table of Contents
- Stack Connectors
- Connector Types
- Developing New Connector Types
Connector Types
Kibana ships with a set of built-in connector types. See Connectors Documentation.
In addition to the documented configurations, several built in connector type offer additional params configurations.
ServiceNow ITSM
The ServiceNow ITSM user documentation params lists configuration properties for the pushToService subaction. In addition, several other subaction types are available.
params
| Property | Description | Type |
|---|---|---|
| subAction | The subaction to perform. It can be pushToService, getFields, getIncident, and getChoices. |
string |
| subActionParams | The parameters of the subaction. | object |
subActionParams (pushToService)
| Property | Description | Type |
|---|---|---|
| incident | The ServiceNow incident. | object |
| comments | The comments of the case. A comment is of the form { commentId: string, version: string, comment: string }. |
object[] (optional) |
The following table describes the properties of the incident object.
| Property | Description | Type |
|---|---|---|
| short_description | The title of the incident. | string |
| description | The description of the incident. | string (optional) |
| externalId | The ID of the incident in ServiceNow. If present, the incident is updated. Otherwise, a new incident is created. | string (optional) |
| severity | The severity in ServiceNow. | string (optional) |
| urgency | The urgency in ServiceNow. | string (optional) |
| impact | The impact in ServiceNow. | string (optional) |
| category | The category in ServiceNow. | string (optional) |
| subcategory | The subcategory in ServiceNow. | string (optional) |
| correlation_id | The correlation id of the incident. | string (optional) |
| correlation_display | The correlation display of the ServiceNow. | string (optional) |
subActionParams (getFields)
No parameters for the getFields subaction. Provide an empty object {}.
subActionParams (getIncident)
| Property | Description | Type |
|---|---|---|
| externalId | The ID of the incident in ServiceNow. | string |
subActionParams (getChoices)
| Property | Description | Type |
|---|---|---|
| fields | An array of fields. Example: [category, impact]. |
string[] |
ServiceNow Sec Ops
The ServiceNow SecOps user documentation params lists configuration properties for the pushToService subaction. In addition, several other subaction types are available.
params
| Property | Description | Type |
|---|---|---|
| subAction | The subaction to perform. It can be pushToService, getFields, getIncident, and getChoices. |
string |
| subActionParams | The parameters of the subaction. | object |
subActionParams (pushToService)
| Property | Description | Type |
|---|---|---|
| incident | The ServiceNow security incident. | object |
| comments | The comments of the case. A comment is of the form { commentId: string, version: string, comment: string }. |
object[] (optional) |
The following table describes the properties of the incident object.
| Property | Description | Type |
|---|---|---|
| short_description | The title of the security incident. | string |
| description | The description of the security incident. | string (optional) |
| externalId | The ID of the security incident in ServiceNow. If present, the security incident is updated. Otherwise, a new security incident is created. | string (optional) |
| priority | The priority in ServiceNow. | string (optional) |
| dest_ip | A list of destination IPs related to the security incident. The IPs will be added as observables to the security incident. | (string | string[]) (optional) |
| source_ip | A list of source IPs related to the security incident. The IPs will be added as observables to the security incident. | (string | string[]) (optional) |
| malware_hash | A list of malware hashes related to the security incident. The hashes will be added as observables to the security incident. | (string | string[]) (optional) |
| malware_url | A list of malware URLs related to the security incident. The URLs will be added as observables to the security incident. | (string | string[]) (optional) |
| category | The category in ServiceNow. | string (optional) |
| subcategory | The subcategory in ServiceNow. | string (optional) |
| correlation_id | The correlation id of the security incident. | string (optional) |
| correlation_display | The correlation display of the security incident. | string (optional) |
subActionParams (getFields)
No parameters for the getFields subaction. Provide an empty object {}.
subActionParams (getIncident)
| Property | Description | Type |
|---|---|---|
| externalId | The ID of the security incident in ServiceNow. | string |
subActionParams (getChoices)
| Property | Description | Type |
|---|---|---|
| fields | An array of fields. Example: [priority, category]. |
string[] |
ServiceNow ITOM
The ServiceNow ITOM user documentation params lists configuration properties for the addEvent subaction. In addition, several other subaction types are available.
params
| Property | Description | Type |
|---|---|---|
| subAction | The subaction to perform. It can be addEvent, and getChoices. |
string |
| subActionParams | The parameters of the subaction. | object |
subActionParams (addEvent)
| Property | Description | Type |
|---|---|---|
| source | The name of the event source type. | string (optional) |
| event_class | Specific instance of the source. | string (optional) |
| resource | The name of the resource. | string (optional) |
| node | The Host that the event was triggered for. | string (optional) |
| metric_name | Name of the metric. | string (optional) |
| type | The type of event. | string (optional) |
| severity | The category in ServiceNow. | string (optional) |
| description | The subcategory in ServiceNow. | string (optional) |
| additional_info | Any additional information about the event. | string (optional) |
| message_key | This value is used for de-duplication of events. All actions sharing this key will be associated with the same ServiceNow alert. | string (optional) |
| time_of_event | The time of the event. | string (optional) |
Refer to ServiceNow documentation for more information about the properties.
subActionParams (getChoices)
| Property | Description | Type |
|---|---|---|
| fields | An array of fields. Example: [severity]. |
string[] |
Jira
The Jira user documentation params lists configuration properties for the pushToService subaction. In addition, several other subaction types are available.
params
| Property | Description | Type |
|---|---|---|
| subAction | The subaction to perform. It can be pushToService, getIncident, issueTypes, fieldsByIssueType, issues, issue, and getFields. |
string |
| subActionParams | The parameters of the subaction. | object |
subActionParams (pushToService)
| Property | Description | Type |
|---|---|---|
| incident | The Jira incident. | object |
| comments | The comments of the case. A comment is of the form { commentId: string, version: string, comment: string }. |
object[] (optional) |
The following table describes the properties of the incident object.
| Property | Description | Type |
|---|---|---|
| summary | The title of the issue. | string |
| description | The description of the issue. | string (optional) |
| externalId | The ID of the issue in Jira. If present, the incident is updated. Otherwise, a new incident is created. | string (optional) |
| issueType | The ID of the issue type in Jira. | string (optional) |
| priority | The name of the priority in Jira. Example: Medium. |
string (optional) |
| labels | An array of labels. Labels cannot contain spaces. | string[] (optional) |
| parent | The ID or key of the parent issue. Only for Sub-task issue types. |
string (optional) |
| otherFields | An object containing key-value pairs of any other fields in Jira without explicit properties. | Record<string, any> (optional) |
subActionParams (getIncident)
| Property | Description | Type |
|---|---|---|
| externalId | The ID of the issue in Jira. | string |
subActionParams (issueTypes)
No parameters for the issueTypes subaction. Provide an empty object {}.
subActionParams (fieldsByIssueType)
| Property | Description | Type |
|---|---|---|
| id | The ID of the issue type in Jira. | string |
subActionParams (issues)
| Property | Description | Type |
|---|---|---|
| title | The title to search for. | string |
subActionParams (issue)
| Property | Description | Type |
|---|---|---|
| id | The ID of the issue in Jira. | string |
subActionParams (getFields)
No parameters for the getFields subaction. Provide an empty object {}.
IBM Resilient
The IBM Resilient user documentation params lists configuration properties for the pushToService subaction. In addition, several other subaction types are available.
params
| Property | Description | Type |
|---|---|---|
| subAction | The subaction to perform. It can be pushToService, getFields, incidentTypes, and `severity. |
string |
| subActionParams | The parameters of the subaction. | object |
subActionParams (pushToService)
| Property | Description | Type |
|---|---|---|
| incident | The IBM Resilient incident. | object |
| comments | The comments of the case. A comment is of the form { commentId: string, version: string, comment: string }. |
object[] (optional) |
The following table describes the properties of the incident object.
| Property | Description | Type |
|---|---|---|
| name | The title of the incident. | string (optional) |
| description | The description of the incident. | string (optional) |
| externalId | The ID of the incident in IBM Resilient. If present, the incident is updated. Otherwise, a new incident is created. | string (optional) |
| incidentTypes | An array with the IDs of IBM Resilient incident types. | number[] (optional) |
| severityCode | IBM Resilient ID of the severity code. | number (optional) |
subActionParams (getFields)
No parameters for the getFields subaction. Provide an empty object {}.
subActionParams (incidentTypes)
No parameters for the incidentTypes subaction. Provide an empty object {}.
subActionParams (severity)
No parameters for the severity subaction. Provide an empty object {}.
Swimlane
Refer to the Run connector API documentation for the full list of properties.
params
| Property | Description | Type |
|---|---|---|
| subAction | The subaction to perform. It can be pushToService. |
string |
| subActionParams | The parameters of the subaction. | object |
subActionParams (pushToService)
| Property | Description | Type |
|---|---|---|
| incident | The Swimlane incident. | object |
| comments | The comments of the case. A comment is of the form { commentId: string, version: string, comment: string }. |
object[] (optional) |
The following table describes the properties of the incident object.
| Property | Description | Type |
|---|---|---|
| alertId | The alert id. | string (optional) |
| caseId | The case id of the incident. | string (optional) |
| caseName | The case name of the incident. | string (optional) |
| description | The description of the incident. | string (optional) |
| ruleName | The rule name. | string (optional) |
| severity | The severity of the incident. | string (optional) |
Ospgenie
Refer to the Run connector API documentation for the full list of properties.
params
| Property | Description | Type |
|---|---|---|
| subAction | The subaction to perform. It can be createAlert or closeAlert. |
string |
| subActionParams | The parameters of the subaction. | object |
subActionParams (createAlert)
| Property | Description | Type |
|---|---|---|
| message | The alert message. | string |
The optional parameters alias, description, responders, visibleTo, actions, tags, details, entity, source, priority, user, and note are supported. See the Opsgenie API documentation for more information on their types.
subActionParams (closeAlert)
No parameters are required. For the definition of the optional parameters see the Opsgenie API documentation.
PagerDuty
The PagerDuty user documentation params lists configuration properties for the params. For more details on these properties, see PagerDuty v2 event parameters .
params
| Property | Description | Type |
|---|---|---|
| eventAction | The type of event. | trigger | resolve | acknowledge |
| dedupKey | All actions sharing this key will be associated with the same PagerDuty alert. This value is used to correlate trigger and resolution. The maximum length is 255 characters. | string |
| summary | An optional text summary of the event. The maximum length is 1024 characters. | string (optional) |
| source | An optional value indicating the affected system, preferably a hostname or fully qualified domain name. Defaults to the Kibana saved object id of the action. | string (optional) |
| severity | The perceived severity of on the affected system. Default: info. |
critical | error | warning | info |
| timestamp | An optional ISO-8601 format date-time, indicating the time the event was detected or generated. | date (optional) |
| component | An optional value indicating the component of the source machine that is responsible for the event, for example mysql or eth0. |
string (optional) |
| group | An optional value indicating the logical grouping of components of a service, for example app-stack. |
string (optional) |
| class | An optional value indicating the class/type of the event, for example ping failure or cpu load. |
string (optional) |
| links | List of links to add to the event | Array<{ href: string; text: string }> (optional) |
| customDetails | Additional details to add to the event. | object |
Developing New Connector Types
When creating a new connector type, your plugin will eventually call server.plugins.actions.setup.registerType() to register the type with the actions plugin, but there are some additional things to think about about and implement.
Consider working with the alerting team on early structure /design feedback of new connectors, especially as the APIs and infrastructure are still under development.
Don't forget to ping @elastic/security-detections-response to see if the new connector should be enabled within their solution.
Licensing
Currently connectors are licensed as "basic" if the connector only interacts with the stack, eg the server log and es index connectors. Other connectors are at least "gold" level.
Plugin location
If the new connector is generic across the stack, it probably belongs in the stack_connectors plugin, but if your connector is very specific to a plugin/solution, it might be easiest to implement it in that plugin/solution.
Connectors that take URLs or hostnames should check that those values are allowed by using the allowed host utilities in the actions plugin.
Documentation
You should create asciidoc for the new connector type. Add an entry to the connector type index - docs/user/alerting/action-types.asciidoc, which points to a new document for the connector type that should be in the directory docs/user/alerting/action-types.
We suggest following the template provided in docs/action-type-template.asciidoc. The Email action type is an example of documentation created following the template.
Tests
The connector type should have both unit tests and functional tests. For functional tests, if your connector interacts with a 3rd party service via HTTP, you may be able to create a simulator for your service to test with. See the existing functional test servers in the directory x-pack/test/alerting_api_integration/common/plugins/actions_simulators/server
Connector type config and secrets
Connector types must define config and secrets which are used to create connectors. This data should be described with @kbn/config-schema object schemas, and you MUST NOT use schema.maybe() to define properties.
This is due to the fact that the structures are persisted in saved objects, which performs partial updates on the persisted data. If a property value is already persisted, but an update either doesn't include the property, or sets it to undefined, the persisted value will not be changed. Beyond this being a semantic error in general, it also ends up invalidating the encryption used to save secrets, and will render the secrets unable to be unencrypted later.
Instead of schema.maybe(), use schema.nullable(), which is the same as schema.maybe() except that when passed an undefined value, the object returned from the validation will be set to null. The resulting type will be property-type | null, whereas with schema.maybe() it would be property-type | undefined.
User interface
To make this connector usable in the Kibana UI, you will need to provide all the UI editing aspects of the connector. The existing connector type user interfaces are defined in x-pack/plugins/triggers_actions_ui/public/application/components/builtin_action_types. For more information, see the UI documentation.