## Summary
#### Security Solution changes:
- Adds new feature flag for `kill-process` operation against SentinelOne
hosts
- Adds support for `kill-process` to the existing api for `agent_type`
of `sentinel_one`
#### Stack Connectors changes:
The following changes were done to the SentinelOne connector:
- Added additional query param to the `getRemoteScripts()` sub-action
## Summary
Fixes#187007
Hides ML embeddables from the "Add panel" flyout when
1. ML feature isn't available for the user role
2. ML is hidden in a current space
### How to test
1. Create a custom role with disabled ML privilege and assign it to a
user
2. Remove ML feature visibility in a current space
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
**Resolves: https://github.com/elastic/kibana/issues/180145**
**Resolves: https://github.com/elastic/kibana/issues/184364**
> [!NOTE]
> This PR doesn't include `isCustomized` recalculation when bulk editing
rules. This should be addressed separately as it might require first
changing the `RulesClient.bulkEdit` method signature.
## Summary
This PR implements the calculation of `ruleSource.isCustomized` inside
the `DetectionRulesClient`. The recalculation of the `isCustomized`
field is performed on every rule patch and update operation, including
rule upgrades. See the ticket for more information:
https://github.com/elastic/kibana/issues/180145 and
`detection_rules_client/mergers/rule_source/calculate_is_customized.ts`
for implementation details.
The `isCustomized` calculation is based on the `calculateRuleFieldsDiff`
method used inside the prebuilt rules domain for calculating changed
fields during rule upgrades. This ensures that the diff calculation
logic is unified and reused to avoid any discrepancies in different
paths of rule management.
The recalculation and saving of the field is done in the following
endpoints:
- **Update Rule** - `PUT /rules`
- **Patch Rule** - `PATCH /rules`
- **Bulk Update Rules** - `PUT /rules/_bulk_update`
- **Bulk Patch Rules** - `PATCH /rules/_bulk_update`
- **Import Rules** - `POST /rules/_import`
- **Perform Rule Upgrade** - `POST /prebuilt_rules/upgrade/_perform`
This PR also partially addresses refactoring mentioned here:
https://github.com/elastic/kibana/issues/184364. Namely:
- Splits the rule converters into smaller single-responsibility
functions.
- Separate methods to convert RuleResponse to AlertingRule and back
- Separate methods to apply rule patches, updates, or set defaults
- Separate case converters
- Migrates methods to work with RuleResponse instead of alerting type
wherever possible.
- Adds new methods for fetching rules by id or rule id and deprecates
the `readRules`. Although new methods are not exposed yet in the public
client interface, this is something that needs to be addressed
separately.
## Summary
Closes - https://github.com/elastic/kibana/issues/172272
The PR adds the degraded Field Table in the Logs Flyout. The accordion
is kept closed by default. For demo purposes below screenshot will show
it expanded
This PR will also fix a very simply Flaky Test -
https://github.com/elastic/kibana/issues/186244
## Pending Items
- [x] Add Locator for Dataset Quality Page
- [x] Add tests
## Demo
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Adds a new functionality in the AI assistant when in dashboards. If the
users ask for a question which will generate a query then then can use
prompts like:
- `Create a visualization from this query and add this to a dashboard`
- `Create a metric from this query and add this to a dashboard`
- ....
### How it works
- It uses the existing functionality of the assistant to create an ES|QL
query (if the generated query is wrong is not part of this PR)
- The LLM returns the query to the new `add_to_dashboard` function and
with the chart type (if the user has added the preference) and the
configuration needed for the ConfigBuilder it creates a Lens embeddable
and adds it to the dashboard.
### How to test
- Go to advanced settings, find the `Observability AI Assistant scope`
setting and change to Everywhere
- Go to a dahsboard (existing or new)
- Ask a question to the AI such as `I want the 95th percentile of ...
from ... index` or `I want the median of butes from the
kibana_sample_data_logs grouped by the top 5 destinations`
- After the ES|QL query has been generated correctly ask AI to create a
chart from this query and add this to the dashboard
### important note
As this is the first real consumer of the build api for ES|QL I have
fixed and various bugs I discovered in the api.
---------
Co-authored-by: Stratoula Kalafateli <stratoula1@gmail.com>
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This PR adds both user and global Artifacts to the Policy Response tree,
which can be found in the fleet's Policy Response flyout as well as in
the Endpoint Details panel. The data needed to render these fields is
already present in the policy_response API response. I don't believe we
need to provide any kind of backward compatibility since
`policy_response` API response seems to be carrying this info as far
back as `7.17.23` stack version (Cloud @ `7.17.23` with FS @ `7.17.23`
and Agent @ `7.17.23`), see screenshot below:
ac1c5048-6e9f-4e9c-b61d-e76a473c83ed
Updated styling
2ac95589-a228-4444-a781-b02055757c3b
Part of https://github.com/elastic/kibana/issues/186574
## Summary
This PR migrates the Encrypted Saved Object Plugin's route handler that
consumes `authc.getCurrentUser` to use `coreStart.security`.
Background: This PR serves as an example of a plugin migrating away from
depending on the Security plugin, which is a high priority effort for
the last release before 9.0.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fixing bug where ad hoc run task was not getting deleted if running the
last schedule entry and the run times out.
## To Verify
1. Create a detection rule
2. Add a delay in the ad hoc task runner
```
--- a/x-pack/plugins/alerting/server/task_runner/ad_hoc_task_runner.ts
+++ b/x-pack/plugins/alerting/server/task_runner/ad_hoc_task_runner.ts
@@ -530,6 +530,7 @@ export class AdHocTaskRunner {
} catch (err) {
runMetrics = asErr(err);
}
+ await new Promise((resolve) => setTimeout(resolve, 3100000));
await this.processAdHocRunResults(runMetrics);
```
3. Schedule a backfill for the rule with only one schedule entry
```
POST https://localhost:5601/internal/alerting/rules/backfill/_schedule
[
{
"rule_id": <ruleId>,
"start": "2024-07-03T13:05:00.000Z"
}
]
```
4. Wait for the run to get cancelled then verify that the task was
deleted.
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
- [x] Shows a banner with the 2024 AVC results blog link in 2 places:
the Security homepage and the Elastic Defend integration page info
- [x] Banner will not show again once dismissed
- [x] Unit Tests
TODO in another pr: have code to remove the avc banner code at the end
of the year
## Screenshots
<img width="1724" alt="image"
src="9ac2ca14-525b-44bc-b357-e87f10856f33">
<img width="1383" alt="image"
src="24ef70fe-dfa7-4fc1-bcba-8405aaf4f9ce">
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Paul Tavares <paul.tavares@elastic.co>
## Summary
Closes https://github.com/elastic/kibana/issues/186873
The ingest pipeline painless scripts for metadata fields were not
looking at the right context fields. This was not a problem until now
because our the fields we collect had the same name in the entity
document (`source field == destination field`), but for collecting
_index we need a different destination to avoid breaking elasticsearch
internals
## Summary
Enables tracing Langchain invocations in the integrations assistant
using the Langsmith settings stored by the Security AI Settings.
The evaluation settings tab is still under an experimental flag, to see
it:
```
xpack.securitySolution.enableExperimental: ['assistantModelEvaluation']
```
### Screenshots
<img width="1317" alt="Settings"
src="6aed1ef6-3750-4259-9fe2-b8bf1aed5504">
After one execution of the integration assistant:
<img width="1240" alt="langsmith"
src="dd3dd99c-7c83-4a35-95b2-789e7a341031">
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Refactors the function argument structure of code used on Kibana server
for Log Rate Analysis from individual arguments to single objects that
contain all options. The options structure looks like this:
```
{
// "meta" args like dependencies, general callbacks etc. on the outer most level
esClient,
abortSignal,
...
// within "arguments" we pass in actual options that necessary for the logic of the function
arguments: {
start,
end,
query,
fields,
...
}
}
```
The main benefit is that code where these functions are used become
easier to read. Instead of the strict order of args that sometimes
included `undefined` or just a value where it's hard to guess for which
argument it's used for, this enforces to have the names of options show
up in the consuming code. Here's an example:
Before:
```
await fetchHistogramsForFields(
client,
requestBody.index,
histogramQuery,
[
{
fieldName: requestBody.timeFieldName,
type: KBN_FIELD_TYPES.DATE,
interval: overallTimeSeries.interval,
min: overallTimeSeries.stats[0],
max: overallTimeSeries.stats[1],
},
],
-1,
undefined,
abortSignal,
stateHandler.sampleProbability(),
RANDOM_SAMPLER_SEED
)
```
After:
```
(await fetchHistogramsForFields({
esClient,
abortSignal,
arguments: {
indexPattern: requestBody.index,
query: histogramQuery,
fields: [
{
fieldName: requestBody.timeFieldName,
type: KBN_FIELD_TYPES.DATE,
interval: overallTimeSeries.interval,
min: overallTimeSeries.stats[0],
max: overallTimeSeries.stats[1],
},
],
samplerShardSize: -1,
randomSamplerProbability: stateHandler.sampleProbability(),
randomSamplerSeed: RANDOM_SAMPLER_SEED,
},
})) as [NumericChartData]
```
### Checklist
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Summary
Adds solution-agnostic components to create hierarchical alerts grouping
UIs, adapting the original implementation from Security Solution.
Closes#184398
## To Verify
For existing usages of the `@kbn/grouping` package: verify that the
grouped UIs work correctly (Security Alerts, Cloud Security Posture).
New alerting UI components: checkout
https://github.com/elastic/kibana/pull/183114 (PoC PR), where the
updated `@kbn/grouping` package and these new components are used in
Observability's main Alerts page.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Gerard Soldevila <gerard.soldevila@elastic.co>
Co-authored-by: Vadim Kibana <82822460+vadimkibana@users.noreply.github.com>
Co-authored-by: Alex Szabo <alex.szabo@elastic.co>
Co-authored-by: Tre <wayne.seymour@elastic.co>
## Summary
Addresses some issues with navigation in the Observability solution
related to onboarding.
### Overview page add data link
### Overview get started link
Doesn't render as an anchor because there's a telemetry call inside the
handler
### Install Metricbeat link
Takes user to overview page pre-filtered for `infra`.
---------
Co-authored-by: Joe Reuter <johannes.reuter@elastic.co>
Part of https://github.com/elastic/kibana/issues/175138
PR decouples "Add to case" action from legacy embeddable framework. PR
also cleans up page load bundle size by moving `isCompatible` and
`execute` implemenations behind async imports.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
As discussed, I looked into making it clearer how to handle the log
stream embeddable panel on dashboards.
It's not possible to show an info icon or a badge without bigger
changes, but there is already a tooltip which can be used for the same
purpose:
<img width="321" alt="Screenshot 2024-07-04 at 11 30 27"
src="60de35b5-559f-4670-b2b9-e074a3cb73c8">
<img width="422" alt="Screenshot 2024-07-04 at 11 31 31"
src="3ba2f87c-dc33-4a6b-bf81-2e561e6b7cec">
I added the "deprecated" to the title instead.
There is code to show a "deprecated" badge, but it only works for
visualization types, not for actions (which is how log stream is
integrated here). Actions currently don't have a notion of deprecation.
It would be possible to add that, but it doesn't seem worth it to
slightly change how the "deprecated" text is rendered.
## Summary
*(Yeah, the title is pretty bad I apologize, I couldn't find something
sexy. OTOH, "sexy" and "logging" are usually antonyms, like "sport car"
and "fiat panda", or "server language" and "javascript")*
### 1. Provide a more developer-friendly alternative to
`Logger.isLevelEnabled`.
**With `isLevelEnabled`**
```ts
if(logger.isLevelEnabled('info')) {
const message = someExpensiveMessageProbablyBasedOnJsonStringifyOrSomething();
logger.info(message);
}
```
**With this PR:**
```ts
logger.info(() => someExpensiveMessageProbablyBasedOnJsonStringifyOrSomething());
```
### 2. Adapt calls to `log.debug` (arguably) costly to use this syntax
Aka any call relying on `JSON.stringify` or function calls.
I used the new syntax for those, except when the tests were too
complicated to fix or when the code did not allow it (e.g. untyped let
variables infered from return from assignations don't play well with
closures)
## Summary
This PR removed almost all API level Units tests from the Server folder
of our code. Server Side code should always be tested using API test,
Functional test and Integration test. Units test only when there is
complex logic which does post processing of data fetched from the DB.
Unnecessary Unit tests on Server side comes with the cost of maintenance
which is not worth. Hence removing them here.
i kept 1 Unit test file which was testing a scenario around
`uncategorizedOnly` scenario which is not exposed via API. I am not sure
why it was added, but then i didn't see any compelling reason to remove
them.
## Summary
There could be
[scenario](https://docs.paloaltonetworks.com/iot/iot-security-api-reference/iot-security-api/get-vulnerability-instances)
where a date can come in as an array -
`"detected_date": [
"2021-04-19T23:59:59"
],`
The `date` processor fails to handle an array with a string.
This PR adds a `script` processor as a pre processor to date processor
to convert the array of string into a string.
Pipeline after the changes
```json
{
"script": {
"tag": "script_convert_array_to_string",
"description": "Ensures the date processor does not receive an array value.",
"lang": "painless",
"source": "if (ctx.palo_alto_iot.vulnerability.detected_date instanceof ArrayList){\n ctx.palo_alto_iot.vulnerability.detected_date = ctx.palo_alto_iot.vulnerability.detected_date[0];\n}\n"
}
},
{
"date": {
"if": "ctx.palo_alto_iot?.vulnerability?.detected_date != null",
"tag": "date_processor_palo_alto_iot.vulnerability.detected_date",
"field": "palo_alto_iot.vulnerability.detected_date",
"target_field": "event.start",
"formats": [
"ISO8601"
]
}
}
```
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Updates to unified field list on typing are debounced - this way we
don't get so many updates when typing in the search input.
Flaky test runner:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6424
## Performance comparison
Test: typing the string: activem for metricbeat data (~6000 fields)
before (costly update on every keystroke):
<img width="669" alt="Screenshot 2024-06-28 at 17 28 38"
src="7075f7bc-2d90-4177-acac-69ac101b2ef1">
after (only one costly update when user stops typing):
<img width="269" alt="Screenshot 2024-06-28 at 17 24 43"
src="8c0ce4a3-7c1a-428b-a482-f6b4d87911e0">
Resolves: #187268
This PR classifies the verification_errors thrown by the Elasticsearch
Query (ES|QL) rules as user error.
## To verify
Create an Elasticsearch Query with ES|QL and use anon-existing field in
the query.
Let the rule run.
`/api/task_manager/metrics?reset=false` should show a user error under
`alerting:__es-query"`
```
"alerting:__es-query":{"success":0,"not_timed_out":1,"total":1,"total_errors":1,"user_errors":1,"framework_errors":0}
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This PR aligns the naming of the entity discovery framework components
to the following pattern:
`entities-{schema version}-{history|latest}-{definition ID}` (with a few
differences here and there, the index has a leading dot, some indices
have a date after them).
---------
Co-authored-by: Chris Cowan <chris@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This PR adds OpenAPI schemas for Defend Workflows API endpoints that
previously didn't have them. Here are the changes made:
1. Added a schema for `/api/endpoint/isolate`, which is deprecated and
now redirects as a `308` to the new path
(`/api/endpoint/action/isolate`). It's tagged with `x-labels` as `ess`
only.
2. Added a schema for `/api/endpoint/unisolate`, which is deprecated and
now redirects as a `308` to the new path
(`/api/endpoint/action/unisolate`). It's tagged with `x-labels` as `ess`
only.
3. Added a schema for
`/api/endpoint/protection_updates_note/{package_policy_id}`.
4. Added `x-labels` field to all existing Defend Workflows API paths for
proper tagging.
For more information on `x-labels`, please refer to
https://github.com/elastic/kibana/pull/184348
## Summary
This PR removes `@timestamp` from the latest index template mappings.
This will help users who explore `.entities-observability.latest-v1.*`
in Discover ES|QL by using the date picker to filter the results.
### Testing
Using the following ES|QL in Discover:
```
FROM .entities-observability.latest-v1.* | LIMIT 10
```
The date picker should look like this:
<img width="932" alt="image"
src="bf75c785-44f6-41b9-b12f-661327012819">
Co-authored-by: Milton Hultgren <milton.hultgren@elastic.co>
This changes the resource names from being stateful to being static.
This makes it easier to import them since they don't have to be passed
around, and "go-to-definition" actually takes you to the implementation
point instead of the types.
## Summary
Ticket https://github.com/elastic/kibana/issues/187384
These changes fix the error on saving the alert
> An error occurred during rule execution: message: "[1:6778] failed to
parse field [kibana.alert.original_event.action] of type [keyword] in
document with id '027b925ae2799635a0dee97a6aa9d58dc87d9771'."
which happens due to not stripping non-ECS compliant sub-fields of the
`event.action` field.
See the main ticket for steps to reproduce the issue.
