Commit graph

884 commits

Author SHA1 Message Date
Patryk Kopyciński
26dd61efa2
BedrockChat & GeminiChat (#186809)
## Summary

Adopted `BedrockChat` from `@langchain/community` package that adds
support for tools calling
https://js.langchain.com/v0.2/docs/integrations/chat/bedrock/

Adopted `ChatGoogleGenerativeAI ` from `@langchain/google-genai` package
that adds support for tools calling
https://js.langchain.com/v0.2/docs/integrations/chat/google_generativeai

Hidden behind FF:
`--xpack.securitySolution.enableExperimental=[assistantBedrockChat]`

As of this PR `integration_assistant` is still going to use
`ActionsClientSimpleChatModel`. After the FF will be enabled by default
we will switch `integration_assistant` to use new chat model.

Thank you @stephmilovic a ton 🙇

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Steph Milovic <stephanie.milovic@elastic.co>
Co-authored-by: Garrett Spong <spong@users.noreply.github.com>
2024-07-23 15:17:21 -05:00
Chris Cowan
cc92c478c8
[Data Forge] Add service.logs dataset as a data stream (#188786)
This PR adds the `service.logs` dataset to Data Forge . The EEM project
needs this dataset to test the default service logs entity definition.
This dataset is different because I wanted to create a fully compliant
data stream. This change also includes changes to the name of the
default ingest pipeline to `logs@custom` to work with the `logs-*-*`
component templates and ingest pipelines. If a document has
`data_stream.dataset` it will now be routed to
`logs-${doc.data_stream.dataset}-default`. If the document has
`data_stream.type`, `data_stream.dataset`, and `data_stream.namespace`
it will be index to `{type}-{dataset}-{namespace}` following the default
data stream conventions.

Because I've changed the name of the ingest pipeline, I also had to
update the index templates for the other datasets.

### Testing

Use the following YAML:

```yaml
---
elasticsearch:
  installKibanaUser: false

kibana:
  installAssets: false
  host: "http://localhost:5601/kibana"

indexing:
  dataset: "service.logs"
  eventsPerCycle: 100
  interval: 10000

schedule:
  - template: "good"
    start: "now-1h"
    end: false
    eventsPerCycle: 100
```

Click on "Logs" under "Observability", you should see something like:

<img width="2048" alt="image"
src="https://github.com/user-attachments/assets/64837c5c-9380-4897-9ccc-acae313683ee">

To check the other data source, change `dataset` to `fake_stack`,
`fake_logs`, `fake_hosts` and check `event.ingested` is set on the
documents; none of these show up in the "Logs Explorer", you'll have to
check them out seperately via "Dev Console".
2024-07-23 12:52:16 -05:00
Walter Rafelsberger
dffc044211
[ML] AIOps: Tweak log rate changes in log rate analysis results table. (#188648)
## Summary

Part of #187684.

This moves functions related to log rate changes to the
`@kbn/aiops_log_rate_analysis` package.

- `getLogRateAnalysisType` was renamed to
`getLogRateAnalysisTypeForHistogram` to indicate its use with histogram
data.
- `getLogRateAnalysisTypeForCounts` was added for cases where we don't
have the histogram data available but just the doc counts for baseline
an deviation time ranges. This isn't used yet as of this PR but will be
in a follow up in combination with the o11y AI assistant.
- `getSwappedWindowParameters` is a helper to consolidate inline code
that's used to swap baseline and deviation when we detected a dip in log
rate.
- Rounding for the log rate change messages was tweaked. Changes below
`10x` will now be rounded to one digit to avoid messages like `1x
increase`.
- Tweaked/Shortened the message for 0 in baseline or deviation to just
`45 up from 0 in baseline` / `down to 0 from 45 in baseline`.

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
2024-07-23 10:25:29 +02:00
Luke G
b9e6e56106
[Security Solution] Align field format with platform spec (#187697)
## Summary

This PR aligns Security Solution field format spec with the platform
one, as described in the field spec.
2024-07-23 09:02:51 +02:00
Chris Cowan
375c6ffd61
[EEM] Convert route validation to Zod (#188691)
## Summary

This PR closes https://github.com/elastic/kibana/issues/188171 by
converting the route validate to Zod for `get`, `reset`, and `delete`
APIs. This also changes the validation for the `create` API to use
`buildRouteValidationWithZod` along with adding `strict()` to each of
the schemas.

Closes https://github.com/elastic/elastic-entity-model/issues/103

---------

Co-authored-by: Kevin Lacabane <kevin.lacabane@elastic.co>
2024-07-22 12:24:29 -05:00
James Gowdy
afe3b0f42c
[ML] AIOps Fixing runtime mappings in pattern analysis (#188530)
Runtime mappings need to be passed to the categorization request factory
function and the field validation function.
Initially they were excluded because we only allow pattern analysis on
text fields and it is not possible to create a text runtime field.
However it is possible to apply a filter which uses a runtime field and
doing so causes pattern analysis to fail.

@walterra I have not investigated log rate analysis' behaviour, in this
PR I have just updated the call to `createCategoryRequest` to pass
`undefined`

To test, create a runtime mapping in the data view. Use this in the
query bar or in a filter in Discover and ML's Log Pattern Analysis page.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-07-20 00:38:19 +10:00
Maxim Kholod
6e73444ca3
[Cloud Security] kick off the work on the DistributionBar component (#188509)
## Summary

Contributes to:
- https://github.com/elastic/security-team/issues/9954

The PR contains the base for the `DistributionBar` component to be used
in the new Entity Flyout Insights.

Not included:
- badges per distribution with the number of documents and pretty names
- on hover interaction

## Screenshots
<img width="980" alt="Screenshot 2024-07-17 at 15 13 48"
src="https://github.com/user-attachments/assets/f2ca53ee-c054-4923-aa3f-7dd4017754cb">

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-07-19 10:38:01 +02:00
Philippe Oberti
1684d37ba2
[Security Solution] - remove old network details flyout and panel (#187870) 2024-07-18 10:35:49 -05:00
Chris Cowan
76f329dfb1
[Data Forge] Restore queue.drain() (#188603)
This PR is a follow up to #187901 – It restores the `await
queue.drain()` function call in the `createEvents()` method. Without the
`queue.drain()` call, the script will run out of memory when indexing
more than 24 hours of data because it will generate events faster than
the queue can drain them and eventually run out of memory.
2024-07-18 08:44:33 -06:00
Milton Hultgren
d5d3f42aa8
[EEM] Fallback to source for metadata if no destination defined (#188515)
This PR closes
https://github.com/elastic/elastic-entity-model/issues/116 by ensuring
that `destination` is always set when the schema is parsed along with
ensuring that if for some reason desitnation is not set, we fallback in
the actual metadata code as well. I also added a unit test for each of
the different `metadata` formats:
- String
- Object with only `source`
- Object with `source` and `limit`
- Object with `source`, `limit`, and `destination`

---------

Co-authored-by: Chris Cowan <chris@elastic.co>
Co-authored-by: Chris Cowan <chris@chriscowan.us>
Co-authored-by: Nathan L Smith <nathan.smith@elastic.co>
2024-07-18 09:51:37 +01:00
Philippe Oberti
62144fbbe1
[Security Solution] - remove old host details flyout and panel (#187831) 2024-07-18 01:59:20 -05:00
Garrett Spong
661c25133d
[Security Assistant] Enables automatic setup of Knowledge Base and LangGraph code paths for 8.15 (#188168)
## Summary

This PR enables the automatic setup of the Knowledge Base and LangGraph
code paths for the `8.15` release. These features were behind the
`assistantKnowledgeBaseByDefault` feature flag, which will remain as a
gate for upcoming Knowledge Base features that were not ready for this
release.

As part of these changes, we now only support the new LangGraph code
path, and so were able to clean up the non-kb and non-RAGonAlerts code
paths. All paths within the `post_actions_executor` route funnel to the
LangGraph implementation.

> [!NOTE]
> We were planning to do the switch to the new
[`chat/completions`](https://github.com/elastic/kibana/pull/184485/files)
public API, however this would've required additional refactoring since
the API's slightly differ. We will make this change and delete the
`post_actions_executor` route for the next release.






### Checklist

Delete any items that are not applicable to this PR.

- [X] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- Working with docs team to ensure updates before merging, cc
@benironside
- [X] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

---------

Co-authored-by: Steph Milovic <stephanie.milovic@elastic.co>
2024-07-17 14:44:24 -07:00
Walter Rafelsberger
c8c27c4820
[ML] AIOps: Chunk groups of field candidates into single queries. (#188137)
## Summary

Part of #187684.

So far we ran individual queries for each field candidate to get
significant items. The historic reason is that we've been uber cautious
not to run into issues with the `max_buckets` setting. But since we
fetch the top 1k items and the `max_buckets` default is 65k it should be
safe to change that.

This PR updates fetching significant items to combine multiple field
candidates within one query using multiple aggs. The current setting in
this PR is now to add up to 50 field candidates into a single query.
This will result in up to ~50k buckets (50 x 1k buckets for the sig
terms agg plus 50 buckets for the cardinality aggs). If there's more
field candidates, we'll still make use of the async queue where we do up
to 5 queries in parallel.

The result is that for example for 200 field candidates we'll just do 4
queries instead of 200 previously.

Previous:

<img width="1624" alt="image"
src="https://github.com/user-attachments/assets/1e11ff1c-a0c2-4dcf-9399-27456439faad">


![aiops-log-rate-analysis-apm-0001](https://github.com/user-attachments/assets/67b6337e-a406-45bc-bb49-85ad047fcbe8)

After:

<img width="1554" alt="image"
src="https://github.com/user-attachments/assets/33ccb9ef-fe5b-4945-a87f-77347ba097ea">


### Checklist

- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
2024-07-17 18:04:39 +02:00
Angela Chuang
60ba001b1f
[Security Assistant] Clean up AI settings tabs (#187705)
## Summary

Code clean up for my previous PR:
1. https://github.com/elastic/kibana/pull/184678#discussion_r1657851993
- Remove additional props.
2. Add `Created on` column for quick prompt and system prompt table
3. Wording change:
https://github.com/elastic/kibana/pull/184678#discussion_r1661034797 -
Rename column title.

**Landing page:**
<img width="1282" alt="Screenshot 2024-07-09 at 19 07 34"
src="20366ee7-497f-412c-9690-953af9f6992b">

**Knowledge base:**
<img width="2552" alt="Screenshot 2024-07-15 at 15 32 40"
src="https://github.com/user-attachments/assets/1d651042-b187-4c08-b55d-c58c1104fd1b">

**Evaluation:**
<img width="2560" alt="Screenshot 2024-07-15 at 15 34 04"
src="https://github.com/user-attachments/assets/31855fe6-e5dd-462d-9c06-2fee2554361a">

<img width="2556" alt="Screenshot 2024-07-09 at 19 38 06"
src="15be4f36-261b-4652-8d4f-be8e7d14676a">

**Anonymization:**
<img width="2551" alt="Screenshot 2024-07-15 at 15 32 33"
src="https://github.com/user-attachments/assets/27688bb5-851e-46fc-8f75-9700ce7a248a">

**Quick prompts:**
<img width="2559" alt="Screenshot 2024-07-15 at 15 30 30"
src="https://github.com/user-attachments/assets/e00c39a0-fb12-46f1-bb2a-bdf5c5bd49d2">

<img width="2557" alt="Screenshot 2024-07-09 at 19 27 18"
src="b581fc46-003b-4363-9c16-22534eb1d71e">

**System prompts:**
<img width="2557" alt="Screenshot 2024-07-15 at 15 30 11"
src="https://github.com/user-attachments/assets/95fd4fca-5041-40b7-b500-efc192166be0">

<img width="2558" alt="Screenshot 2024-07-09 at 19 10 36"
src="a701391a-978f-4684-a2ea-f72a5f572217">

**Conversations:**
<img width="2553" alt="Screenshot 2024-07-15 at 15 30 01"
src="https://github.com/user-attachments/assets/3411beb8-4775-4ba7-8b3e-c4111497eed2">

<img width="2554" alt="Screenshot 2024-07-09 at 21 33 37"
src="fbe2ee80-ba20-41b6-b224-3e317dc1c20e">

Connectors:
<img width="2558" alt="Screenshot 2024-07-09 at 19 09 15"
src="c711ce09-65c0-45b3-90c1-a9019d35093c">




[Design](https://www.figma.com/design/BMvpY9EhcPIaoOS7LSrkL0/[8.15]-GenAI-Security-Assistant-Settings%3A-Stack-Management-Pages?node-id=51-25207&t=JHlgCm0sCYsl8WCM-0)

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
2024-07-17 13:14:21 +01:00
Philippe Oberti
0b8e9bc8fe
[Security Solution] - remove old user details flyout and panel (#187829) 2024-07-16 06:19:46 -05:00
Kevin Lacabane
393375ad67
[EEM] Update the builtin service entity definition (#187021)
## Summary

This change includes updates to the builtin service definition:
- removal of the high cardinality metadata fields until we have a
solution in place
- fetching of the metrics from the preaggregated apm metrics
- fixed metrics aggregations
- increased history transform frequency and delay to cover for delayed
ingestion

---------

Co-authored-by: Milton Hultgren <milton.hultgren@elastic.co>
Co-authored-by: Søren Louv-Jansen <sorenlouv@gmail.com>
2024-07-16 11:45:45 +02:00
Sergi Massaneda
02d5176b72
[Security GenAI] Make langsmith UI settings take precedence over env vars. (#188297) 2024-07-16 09:19:45 +02:00
Patryk Kopyciński
87f0e71d19
Fix support for IME in Assistant prompt (#184874)
## Summary

Kudos to @sakurai-youhei for providing a fix 🙇
 
Tested on Chrome, Firefox, Safari


f0ff388f-3943-4382-a873-88949b760629

---------

Co-authored-by: Garrett Spong <spong@users.noreply.github.com>
2024-07-16 01:21:59 +01:00
Steph Milovic
d5843b351e
[Security solution] Fix LangGraph stream with SimpleChatModel (#187994) 2024-07-15 22:35:42 +01:00
Chris Cowan
2fac5e8462
[Data Forge] Add artificial delay feature (#187901)
## Summary

This PR adds a new setting, `indexing.artificialIndexDelay`, to the
indexing configuration to control how much artificial delay to add to
the timestamps. This PR also adds a "final" ingest pipeline to each data
source along with injecting a new base `component_template` which
includes the `event.ingested` field.

The artificial delay is useful for testing transforms on data that has a
significant delays. It also allows us to test if we miss data when
syncing on the transforms using `event.ingested`.

- Installs default ingest pipeline to add event.ingested to each
document
- Adds final_pipeline to each install_index_template
- Inject base component_template to each index_template at install time
- Add artificial delay for "current" events, historical events are
ingested without delay.
- Change index math to produce monthly indices

### How to test:

Copy the following to `fake_logs.delayed.yaml`:

```YAML
---
elasticsearch:
  installKibanaUser: false

kibana:
  installAssets: true
  host: "http://localhost:5601/kibana"

indexing:
  dataset: "fake_logs"
  eventsPerCycle: 100
  artificialIndexDelay: 300000

schedule:
  - template: "good"
    start: "now-1h"
    end: false
    eventsPerCycle: 100
```
Then run `node x-pack/scripts/data_forge.js --config
fake_logs.delayed.yaml`. This should index an hour of data immediately,
then add a 300s delay when indexing in "real time". The logs will look
like:

```
 info Starting index to http://localhost:9200 with a payload size of 10000 using 5 workers to index 100 events per cycle
 info Installing index templates (fake_logs)
 info Installing components for fake_logs (fake_logs_8.0.0_base,fake_logs_8.0.0_event,fake_logs_8.0.0_log,fake_logs_8.0.0_host,fake_logs_8.0.0_metricset)
 info Installing index template (fake_logs)
 info Indexing "good" events from 2024-07-09T16:23:36.803Z to indefinitely
 info Delaying 100 by 300000ms
 info Waiting 60000ms
 info { took: 2418721239, latency: 541, indexed: 6000 } Indexing 6000 documents.
...
```
Then after `300s`, it will index another `100` documents every `60s`.
You can also inspect the delay per minute using the following ES|QL in
Discover:
```
FROM kbn-data-forge-fake_logs.fake_logs-* | eval diff=DATE_DIFF("seconds", @timestamp, event.ingested) | STATS delay=AVG(diff) by timestamp=BUCKET(@timestamp, 1 minute)
```
This should give you a chart that looks something like this:

<img width="1413" alt="image"
src="2f48cb85-a410-487e-8f3b-41311ff95186">


There should also be a 5 minute gap at the end in Discover:

<img width="1413" alt="image"
src="660acc87-6958-4ce9-a544-d66d56f805dd">

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-07-15 17:49:34 +01:00
Walter Rafelsberger
a87d6ccb1a
[ML] AIOps: Consolidate code related to retrieving log rate analysis parameters from alert metadata. (#188012)
## Summary

Part of #178613.

Consolidates code related to retrieving log rate analysis parameters
from alert metadata.
2024-07-15 14:35:42 +02:00
Yuliia Naumenko
1d5cf48a97
[Security AI Assistant] Changing chat related internal APIs to become public (#188139)
## Summary

In this PR Security Gen AI related APIs are changed from internal to
public.

Conversations APIs:
- POST/PUT/GET/DELETE
`"/api/security_ai_assistant/current_user/conversations/{id}"`
- GET `"/api/security_ai_assistant/current_user/conversations/_find"`

Prompts APIs:

- POST `"/api/security_ai_assistant/prompts/_bulk_action"`
- GET `"/api/security_ai_assistant/current_user/conversations/_find"`

Anonymization APIs:

- POST `"/api/security_ai_assistant/anonymization_fields/_bulk_action"`
- GET `"/api/security_ai_assistant/anonymization_fields/_find"`
2024-07-12 12:23:09 -07:00
Sander Philipse
460b52077f
[Semantic text] Make semantic text work with non-root level fields (#187154)
## Summary

This makes semantic text work with non-root level reference fields. It
also correctly adds copy_to to existing copy_to fields instead of
replacing them, and streamlines a lot of the code.

To test these changes:

- Create an index
- Go to the index mappings page at
`app/management/data/index_management/indices/index_details?{yourIndexName}=blah&tab=mappings`
- Add an object field with a text field inside
- Add a semantic text field referencing that text field
- If you're on a Macbook, create a new inference endpoint with the model
`.elser_model_2` instead of using the default inference endpoint.
- Add a second semantic text field referencing that text field
- Save your mappings
- Use JSON view to verify that the newly created text field contains a
`copy_to` field referencing both newly created semantic text fields
- Verify that the newly created semantic text fields are also in the
JSON view



### Checklist

Delete any items that are not applicable to this PR.

- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [x] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [x] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [x] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
2024-07-11 20:35:56 +10:00
Saikat Sarkar
ff651f20d2
[Inference Endpoints View] Deletion, search and filtering of inference endpoints (#186206)
This PR consists of the following changes:
- An option to delete an existing inference endpoint
- Filtering the endpoints based on 'provider' and 'type'
- Search option
- Display the trained models deployment status
- Display additional 3rd party providers (Mistral, Azure OpenAI, Azure
AI Studio)
- Add licensing for gating enterprise licensed users

### Stack Management
![Screenshot 2024-06-24 at 2 38
44 PM](d8072069-2309-40b9-a723-6b34f64b7ef0)



### Serverless
![Screenshot 2024-06-24 at 2 43
36 PM](fe5be2fd-d9ca-41f7-b246-8767e88d2938)

---------

Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
2024-07-09 08:42:52 -06:00
Milton Hultgren
47178a776f
[EEM] Add versioning for entity definitions (#187692)
This PR adds a `version` field to the `EntityDefinition` type, making it
required in the API calls. It must be a SemVer string.
The version is added to the ingest pipelines and transforms as part of
their metadata.
The version is included in the output documents alongside the schema
version.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-07-09 16:31:35 +02:00
Sid
ff9a48edbe
[Security in Core] Exposes apiKeys from core.security.authc (#186910)
## Summary
Expose `apiKeys` as a service under `core.security.authc.apiKeys`.

Closes https://github.com/elastic/kibana/issues/184764

### Details
PR introduces a new API Keys Service which is accessible under the
`authc` namespace in core.security. The service exposes the public API
that was already available on the server-side in the security plugin.

The service is initialized and registered with core using the
`delegate_api` - allowing access to the service within the core plugin
without the need for the `security` plugin.

Note: I had to move quite a few types/functions around to prevent
cyclical dependencies.

### Plugins and the APIs that use the current `apiKeys` function from
the security plugin
<details>
<summary> Expand for table with details </summary>

| Plugin | File | API used | Can be migrated |
|--------|--------|--------|--------|
| alerting | x-pack/plugins/alerting/plugin/server.ts |
areApiKeysEnabled() |  |
| | x-pack/plugins/alerting/server/rules_client_factory.ts |
grantAsInternalUser() |  |
| | x-pack/plugins/alerting/server/task.ts | invalidatedAsInternalUser()
|  |
| enterprise_search |
x-pack/plugins/enterprise_search/server/routes/enterprise_search/api_keys
| create() |  |
| |
x-pack/plugins/enterprise_search/server/lib/indices/create_api_key.ts |
create() |  |
| fleet | x-pack/plugins/fleet/server/routes/setup/handlers.ts |
areApiKeysEnabled() |  |
| | x-pack/plugins/fleet/server/services/api_keys/security |
invalidateAsInternalUser() |  |
| | x-pack/plugins/fleet/server/services/api_keys/transform_api_keys.ts
| grantAsInternalUser() |  |
| |
x-pack/plugins/fleet/server/services/setup/fleet_server_policies_enrollment_keys.ts
| areApiKeysEnabled() |  |
| |
x-pack/plugins/fleet/server/services/setup/fleet_server_policies_enrollment_keys.ts
| areAPIKeysEnabled() |  |
| |
x-pack/plugins/observability_solution/apm/server/routes/agent_keys/get_agent_keys_privileges.ts
| areAPIKeysEnabled() |  |
| observability_solution |
x-pack/plugins/observability_solution/entity_manager/server/lib/auth/api_key/api_key.ts
| areAPIKeysEnabled |  |
| | | validate |  |
| | | grantAsInternalUser |  |
| |
x-pack/plugins/observability_solution/entity_manager/server/routes/enablement/disable.ts
| invalidateAsInternalUser |  |
| |
x-pack/plugins/observability_solution/entity_manager/server/routes/enablement/enable.ts
| invalidateAsInternalUser |  |
| |
x-pack/plugins/observability_solution/observability_onboarding/server/routes/flow/route.ts
| create |  |
| |
x-pack/plugins/observability_solution/synthetics/server/routes/synthetics_service/enablement.ts
| invalidateAsInternalUser |  |
| |
x-pack/plugins/observability_solution/synthetics/server/synthetics_service/get_api_key.ts
| validate |  |
| | | areAPIKeysEnabled |  |
| | | grantAsInternalUser |  |
| | | create |  |
| serverless_search |
x-pack/plugins/serverless_search/server/routes/api_key_routes.ts |
create |  |
| |
x-pack/plugins/transform/server/routes/api/reauthorize_transforms/route_handler_factory.ts
| grantAsInternalUser |  |
| |
x-pack/plugins/upgrade_assistant/server/lib/reindexing/credential_store.ts
| grantAsInternalUser |  |
| | | invalidateAsInternalUser |  |
| | | areAPIKeysEnabled() |  |
</details>

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2024-07-09 23:43:17 +10:00
Patryk Kopyciński
27ccd4d539
Cleanup aiAssistantFlyoutMode feature flag (#182992)
## Summary

Cleanup Security `aiAssistantFlyoutMode` feature flag

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-07-09 14:57:59 +02:00
Sergi Massaneda
92099b277d
[Security GenAI] Use AI setting to set langsmith tracing to the Integration Assistant (#187466)
## Summary

Enables tracing Langchain invocations in the integrations assistant
using the Langsmith settings stored by the Security AI Settings.
The evaluation settings tab is still under an experimental flag, to see
it:

```
xpack.securitySolution.enableExperimental: ['assistantModelEvaluation']
```

### Screenshots

<img width="1317" alt="Settings"
src="6aed1ef6-3750-4259-9fe2-b8bf1aed5504">

After one execution of the integration assistant:

<img width="1240" alt="langsmith"
src="dd3dd99c-7c83-4a35-95b2-789e7a341031">

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2024-07-08 20:14:57 +02:00
Walter Rafelsberger
94cab93977
[ML] AIOps: Refactors function argument structure for Log Rate Analysis. (#187669)
## Summary

Refactors the function argument structure of code used on Kibana server
for Log Rate Analysis from individual arguments to single objects that
contain all options. The options structure looks like this:

```
{
  // "meta" args like dependencies, general callbacks etc. on the outer most level
  esClient,
  abortSignal,
  ...
  // within "arguments" we pass in actual options that necessary for the logic of the function
  arguments: {
    start,
    end,
    query,
    fields,
    ...
  }
}
```

The main benefit is that code where these functions are used become
easier to read. Instead of the strict order of args that sometimes
included `undefined` or just a value where it's hard to guess for which
argument it's used for, this enforces to have the names of options show
up in the consuming code. Here's an example:

Before:

```
await fetchHistogramsForFields(
                client,
                requestBody.index,
                histogramQuery,
                [
                  {
                    fieldName: requestBody.timeFieldName,
                    type: KBN_FIELD_TYPES.DATE,
                    interval: overallTimeSeries.interval,
                    min: overallTimeSeries.stats[0],
                    max: overallTimeSeries.stats[1],
                  },
                ],
                -1,
                undefined,
                abortSignal,
                stateHandler.sampleProbability(),
                RANDOM_SAMPLER_SEED
              )
```

After:

```
                (await fetchHistogramsForFields({
                  esClient,
                  abortSignal,
                  arguments: {
                    indexPattern: requestBody.index,
                    query: histogramQuery,
                    fields: [
                      {
                        fieldName: requestBody.timeFieldName,
                        type: KBN_FIELD_TYPES.DATE,
                        interval: overallTimeSeries.interval,
                        min: overallTimeSeries.stats[0],
                        max: overallTimeSeries.stats[1],
                      },
                    ],
                    samplerShardSize: -1,
                    randomSamplerProbability: stateHandler.sampleProbability(),
                    randomSamplerSeed: RANDOM_SAMPLER_SEED,
                  },
                })) as [NumericChartData]
```


### Checklist

- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
2024-07-08 20:06:40 +02:00
Pierre Gayvallet
b6fcfac9c1
[Kibana logging system] Add conditional evaluation based on level for logging APIs (#187225)
## Summary

*(Yeah, the title is pretty bad I apologize, I couldn't find something
sexy. OTOH, "sexy" and "logging" are usually antonyms, like "sport car"
and "fiat panda", or "server language" and "javascript")*

### 1. Provide a more developer-friendly alternative to
`Logger.isLevelEnabled`.


**With `isLevelEnabled`**
```ts
if(logger.isLevelEnabled('info')) {
  const message = someExpensiveMessageProbablyBasedOnJsonStringifyOrSomething(); 
  logger.info(message);
}
```

**With this PR:**
```ts
logger.info(() => someExpensiveMessageProbablyBasedOnJsonStringifyOrSomething());
``` 

### 2. Adapt calls to `log.debug` (arguably) costly to use this syntax

Aka any call relying on `JSON.stringify` or function calls.

I used the new syntax for those, except when the tests were too
complicated to fix or when the code did not allow it (e.g. untyped let
variables infered from return from assignations don't play well with
closures)
2024-07-08 23:53:02 +10:00
Luke G
d823d5deef
[Security Solution] Reduce BrowserField to field spec (#187263)
## Summary

BrowserField is now FieldSpec, with an exception of one field. ("format")
2024-07-05 02:25:07 +10:00
Tom Myers
6ca9fec2e7
remove 'spaceId' from entity APIs (#187502)
remove 'spaceId' from entity APIs
2024-07-04 11:03:24 +01:00
Yuliia Naumenko
0a0bb1498e
[Security AI Assistant] Persist prompts (#187040)
Moving prompts persistence layer from the local storage to the server
side data stream `.kibana-elastic-ai-assistant-prompts`

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-07-04 03:28:15 +10:00
Tom Myers
19db776f4f
make EntityDefinition 'type' a plain string instead of an enum (#187451)
make EntityDefinition 'type' a plain string instead of an enum
2024-07-03 13:15:30 +01:00
Steph Milovic
ad4fe84078
[Security solution] Assistant race condition bug fixing (#187186) 2024-07-02 13:42:22 -06:00
Kurt
4554b75899
Add license check for FIPS (#181187)
## Updates
### Latest updates

- Expose whether KB is configured to run in FIPS mode from Core ->
Security
<img width="653" alt="Screenshot 2024-06-20 at 9 55 17 PM"
src="56a9f50f-0a05-41ca-9292-ed225b3d8062">



Consolidating all FIPS PRs into this PR

*Previous PRs were Approved

### Changes

- Config option is now experimental:
`xpack.security.experimental.fipsMode.enabled`
- Documentation has been revised
  - Listed as an experimental feature
  - Added keystore references for adding a password

## Summary
Closes #169738
Closes #169739
Closes #169740
Closes #185948

FIPS is a platinum license feature. 

KIbana instances must have a platinum or better license to start up in
FIPS mode, a lesser license will result in Kibana failing to start up

If the license is degraded, Kibana will still run, but an error will be
logged letting the user know that Kibana will not be able to restart.

## Config changes

This PR required the changes that were approved from [a previous
PR](https://github.com/elastic/kibana/pull/174558), since that PR
couldn't be merged into main, I merged it here.

## Testing

### Locally

In your `kibana.dev.yml` add:
`xpack.security.experimental.fipsMode.enabled: true`

To allow Kibana to start without actually providing a compliant OpenSSL
provider, in `x-pack/plugins/security/server/config.ts` change L328 from
`if (isFipsEnabled !== isNodeRunningWithFipsEnabled)` to `if (false)`

You are now configured to run in FIPS-spoof mode!

Run: `yarn es snapshot` and `yarn start` > You should see Kibana fail to
start with an error about using a basic license.

Run: `yarn es snapshot --license trial` and `yarn start` > Kibana should
start.

Login as `elastic` and navigate to Stack Management > License Management

Switch your license to `basic` and accept.

In your logs, you will see an error letting users know that you no
longer have an appropriate license and Kibana will not restart.


### For FIPS enthusiasts

Start an ES instance in a method of your choosing, but not using `yarn
es snapshot`. I like to use an 8.15.0-snapshot from the `.es/cache`
directory by running `tar -xzvf
elasticsearch-8.15.0-SNAPSHOT-darwin-aarch64.tar.gz ` and cd into the
new directory's `bin` folder to run `./elasticsearch`

Ensure you have Docker running locally.

From any command line, run: `docker run --rm -it -e
XPACK_SECURITY_FIPSMODE_ENABLED='true' -p 5601:5601/tcp
docker.elastic.co/kibana-ci/kibana-ubi-fips:8.15.0-SNAPSHOT-bc3150316ed317c08d57c6bd785ba39586072e1d`

This will start Kibana into Interactive Setup mode, copy and paste the
token from the ES startup logs.

Kibana should fail to start and you should see Kibana fail to start with
an error about using a basic license.

Repeat the above process except before you paste the token from ES, do
the following to enable a trial license on your ES instance:

In a new terminal window, navigate to your the top level of your
elasticsearch folder and run

`curl -X POST --cacert config/certs/http_ca.crt -u
elastic:YOUR_PASSWORD_HERE
"https://localhost:9200/_license/start_trial?acknowledge=true&pretty"`

You should receive a successful response.

Now paste the token from the ES startup logs into the Kibana Interactive
Setup window and Kibana should start.

Login as `elastic` and navigate to Stack Management > License Management

Switch your license to `basic` and accept.

In your logs, you will see an error letting users know that you no
longer have an appropriate license and Kibana will not restart.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: lcawl <lcawley@elastic.co>
2024-07-02 11:03:28 -04:00
Quynh Nguyen (Quinn)
de027b80b2
[ML] Add Field statistics embeddable as panel in Dashboard (#184030)
## Summary

This PR adds Field statistics embeddable as panel in Dashboard 

By default, it will enable the ES|QL editor for the field stats panel.
It will allow for editing of the ES|QL query, and time range.




4b5438c7-051f-4627-aab1-b802c23ca652



e9bae0e4-17cf-4a86-ad70-0da9d3667b53




If and only if ES|QL is disabled, it will show the data view picker as a
fallback.





### Checklist

Delete any items that are not applicable to this PR.

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)


### Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.

When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:

| Risk | Probability | Severity | Mitigation/Notes |

|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces&mdash;unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes&mdash;Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |


### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
2024-07-01 23:09:44 -05:00
Yuliia Naumenko
d5a91fcc5d
[Security AI Assistant] Chat complete API (#184485) 2024-07-01 22:17:07 -05:00
Vitalii Dmyterko
3506e1413d
[Security Solution][Detection Engine] adds AI Assistant to rule create form (#179091)
## Summary

- adds AI assistant for queries for every rule type, apart Machine
Learning
- AI assistant is shown only when query is not empty and invalid
- When user clicks on assistant it records telemetry event
`open_assistant_on_rule_query_error `
- hidden behind `AIAssistantOnRuleCreationFormEnabled` feature flag


## Design

[Design](https://www.figma.com/file/nbgUduTmTpYNXLf1vDMP7u/General-Enhancements?type=design&node-id=115%3A5166&mode=design&t=2Yi5wvS1aDoYxuyT-1)


![AI assistant help
contextual](c2ad0989-dd1a-4082-be83-bed7741131be)


## Demo



92435f3b-c51e-471b-940f-604a1f245e94



## Old Demoes

**Note: old demo videos use old UI design, and assistant is shown even
for valid queries.**

<details>

<summary>list of videos</summary>

### ES|QL Case 1
Simple ES|QL query validation error solving
There 2 problems in query highlighted by validation.
First, missing metadata operator
Second, operator `=` instead of `==`
By feeding query twice in Ai Assistant, I was able to get working
solution


1eb49505-b161-4fdb-ac3c-d2833c16e2cd

### ES|QL Case 2

Fixes missing _id field, when metadata operator is present



82024fcb-822e-46f1-a80a-8b9f1725816e

### EQL Case 1

fixes EQL typo



ea18ceec-92f8-4322-b359-50e689a0ef72

</details>

### Issues

Results might not be always consistent and for more complex queries they
might not correct



e3bedfd6-943c-4979-8708-f6c33d1756a6

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2024-07-01 12:48:48 -05:00
Agustina Nahir Ruidiaz
9785ce0e59
[Security Solution] Adding accordion to dashboard security views (#186465)
## Summary

In this feature the following UI changes are required:

Add an accordion layout for wrap all the image link cards.
Change Layout of image cards.
Remove "Default" Header
Change second header label

<img width="1431" alt="Screenshot 2024-06-19 at 15 18 37"
src="df4b9582-d93b-45c9-89bb-3c2e52a01320">


### Checklist

Delete any items that are not applicable to this PR.

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Angela Chuang <6295984+angorayc@users.noreply.github.com>
2024-07-01 09:45:18 -05:00
Sergi Massaneda
51c7afd701
[GenAI] Add License/productType controls to the integration assistant (#187105)
## Summary

Adds the licence and product type controls to the Integration assistant.

### UI

- ESS 

Licence: The `Enterprise` license must be active to use the feature. It
is checked inside the _integration_assistant_ plugin itself by default
and the license paywall is displayed in case the license requirement is
not met, no need to configure anything from outside for the license
check.

- Serverless

Product type: The `Security complete` tier must be enabled to use the
feature in security projects. The _integration_assistant_ plugin exposes
a contract API to set up a custom upselling.
The product type check is performed in the
_security_solution_serverless_ plugin and the upselling component is
passed to the _integration_assistant_ plugin to be rendered.

### API

The `withAvailability` wrapper has been introduced to encapsulate this
availability check on all the routes. The `isAvailable` flag is defined
at a plugin level and passed to the router context. The flag is
defaulted to `true` and can be set to `false` by not having the
`Enterprise` license (ESS), or by calling the `setIsActive(false)`
contract exposed (serverless).

All API requests done while the license / product type requirements are
not met will be responded with:
```
404 Not Found: This API route is not available using your current license/tier.
```


### Screenshots

ESS:


![ess](e3f4947e-4f86-4536-bdc0-a14715c01d2f)

Serverless:


![serverless](d7ba23d9-cb4e-429a-9dc8-034840afdde7)

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-07-01 08:43:35 -05:00
Steph Milovic
92b2ec62d6
[Security solution] Attack discovery connector dropdown notification badges (#187209) 2024-06-29 13:47:42 -06:00
Jonathan Budzenski
1536a4e607 Revert "[Security solution] Attack discovery connector dropdown notification badges (#186903)"
This reverts commit b131161848.
2024-06-29 12:47:46 -05:00
Steph Milovic
b131161848
[Security solution] Attack discovery connector dropdown notification badges (#186903) 2024-06-28 19:08:22 -06:00
Angela Chuang
72e1d11b25
[Security Solution] AI settings (#184678)
## Summary
https://github.com/elastic/security-team/issues/9222
<img width="2535" alt="Screenshot 2024-06-23 at 11 30 15"
src="e47c4cc4-6786-4147-a6f8-e3b371d64e40">
<img width="2534" alt="Screenshot 2024-06-23 at 11 30 54"
src="7d6f8f17-a713-4c38-8a2e-369f8548426e">
<img width="2537" alt="Screenshot 2024-06-23 at 11 37 11"
src="c6632fff-0d01-4462-b5f1-c4dc3b2750dc">
<img width="2531" alt="Screenshot 2024-06-25 at 13 36 19"
src="d3d6f6a1-02d5-40cc-9119-09de112d53cd">

<img width="2531" alt="Screenshot 2024-06-25 at 13 33 59"
src="742a57ff-9ec0-4fe6-868e-810681ff1795">
<img width="2534" alt="Screenshot 2024-06-23 at 11 40 31"
src="85e323d2-6fdf-40cd-9696-4c8c549200a3">
<img width="2535" alt="Screenshot 2024-06-23 at 11 41 23"
src="e02fdee2-dd57-45b6-8fef-b75f535c96fc">

Knowledge base:
https://github.com/elastic/kibana/pull/186847


### Checklist

Delete any items that are not applicable to this PR.

- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
2024-06-28 21:30:03 +01:00
Christiane (Tina) Heiligers
a7fe7cbb52
update imports for types moved to core security service packages (#187121)
## Summary

Part of https://github.com/elastic/kibana/issues/186574

Updates imports for types available from the core-security packages.

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2024-06-28 10:13:21 -07:00
Luke G
6e6dff0758
Remove example and description from browser field (#186897)
## Summary

Further progress towards cleaning up the sourcerer model, aligning it
with the FieldSpec. This PR removes description and examples fields from
the `BrowserField`, sourcing these from the EcsFlat available in
`@elastic/ecs` package.
2024-06-27 10:21:34 +02:00
Pierre Gayvallet
1dee2872f3
Bump @elastic/elasticsearch to 8.14.0 (reloaded) (#186848)
## Summary

Reopening https://github.com/elastic/kibana/pull/186326 with my account,
non-internal PRs are just terrible to work with

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@elastic.co>
2024-06-26 08:29:15 -07:00
Luke G
ac9f6233eb
remove category from the browser field (#186839)
## Summary

Cleaning up the sourcerer model a bit, `category` does not seem to be
used anywhere and it is one of the things that deviate from the
FieldSpec.
2024-06-26 12:05:10 +02:00
Steph Milovic
48c0e0dd7c
[Security solution] Attack discovery background task and persistence (#184949) 2024-06-25 08:43:39 -07:00