Resolves https://github.com/elastic/response-ops-team/issues/221
## Summary
Looking through the logs for this, the majority of these errors have the
following stack trace (14,000+ in the last 7 days):
```
ResponseError: search_phase_execution_exception
Root causes:
no_shard_available_action_exception: [es-es-search-686bf6b747-xldl8][100.65.77.183:9300][indices:data/read/search[phase/query]]
no_shard_available_action_exception: null
at KibanaTransport._request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:543:27)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at /usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:641:32
at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:637:20)
at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:60:16)
at ClientTraced.SearchApi [as search] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/search.js:72:12)
at getTotalAlertsCountAggregations (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/usage/lib/get_telemetry_from_alerts.js:42:21)
at async Promise.all (index 6)
at TaskManagerRunner.run (/usr/share/kibana/node_modules/@kbn/task-manager-plugin/server/task_running/task_runner.js:325:22)
```
Looking through the code, we are already catching these errors and
returning a default response for that telemetry object. The
`no_shard_available_action_exception` is not an issue with Kibana, it's
an ES issue, so this PR catches these types of errors and log them at a
debug level instead of a warn level to avoid polluting the logs with
errors we have no control over.
Excluding those results, we see a different and less frequent stack
trace (100+ in the last 15 days):
```
TypeError: Cannot read properties of undefined (reading 'by_rule_type_id')
at getTotalAlertsCountAggregations (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/usage/lib/get_telemetry_from_alerts.js:49:109)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at async Promise.all (index 6)
at TaskManagerRunner.run (/usr/share/kibana/node_modules/@kbn/task-manager-plugin/server/task_running/task_runner.js:325:22)
```
For actions telemetry, the volume of these errors is much lower.
Unfortunately, we are not logging the stack trace for these errors so
it's harder to track down the source. I've updated the code to store the
stack trace as well as adding the same handling for
`no_shard_available_action_exception` under the assumption that it could
also be getting these errors.
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
As part of our effort to harden API action definitions and enforce
standards this PR adds an utility `ApiPrivileges` class.
It is supposed to be used for both feature registration and API route
definition to construct the privilege name.
```ts
plugins.features.registerKibanaFeature({
privileges: {
all: {
app: [...],
catalogue: [...],
api: [ApiPrivileges.manage('subject_name')],
...
},
read: {
...
api: [ApiPrivileges.read('subject_name')],
...
},
},
})
....
// route definition
router.get(
{
path: 'api_path',
security: {
authz: {
requiredPrivileges: [ApiPrivileges.manage('subject_name')],
},
},
},
async (ctx, req, res) => {}
);
```
`require_kibana_feature_privileges_naming` eslint rule has been added to
show warning if the API privilege name doesn't satisfy the naming
convention.
### Naming convention
- API privilege should start with valid `ApiOperation`: `manage`,
`read`, `update`, `delete`, `create`
- API privilege should use `_` as separator
❌ `read-entity-a`
❌ `delete_entity-a`
❌ `entity_manage`
✅ `read_entity_a`
✅ `delete_entity_a`
✅ `manage_entity`
> [!IMPORTANT]
> Serverless ZDT update scenario:
>
> - version N has an endpoint protected with the `old_privilege_read`.
> - version N+1 has the same endpoint protected with a new
`read_privilege`.
>
> There might be a short period between the time the UI pod N+1 passes
SO migrations and updates privileges and the time it's marked as
ready-to-handle-requests by k8s, and when UI pod N is terminated.
>
> After discussion with @legrego and @azasypkin we decided to ignore it
due to the perceived risk-to-cost ratio:
> 1. The time window users might be affected is very narrow because we
register privileges late in the Kibana startup flow (e.g., after SO
migrations).
> 2. The transient 403 errors users might get won't result in session
termination and shouldn't lead to data loss.
> 3. The roll-out will be performed in batches over the course of
multiple weeks and implemented by different teams. This means the impact
per release shouldn't be significant.
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
__Relates: https://github.com/elastic/kibana/issues/198716__
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Fixes the bug of the invalid event filter created automatically when
creating a cloud workloads endpoint integration. The issue was a type
issue: `undefined` or an object is expected, instead an array was
passed.
To make sure this does not happen again, the type for the `meta` field
was updated from the deprecated `t.object` to `t.UnknownRecord`, which
is able to catch similar issues as a type error:
ca0c01b63b
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Filters out any deprecated Kibana API usages from blocking upgrade
status.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### Risks
Should be mitigated by E2E tests
Fixes#200772
## 🐉 Summary
This PR fixes supporting data view runtime fields during rule execution
for the custom threshold rule.
## 🧪 How to test
1. Create a runtime field as shown below:
|Runtime field| Preview|
|---|---|
||
2. Make sure alerts are generated as expected both for regular and
no-data alerts:
### TODO
- [x] Add an API integration test
- [x] Test on MKI
## Summary
Part of https://github.com/elastic/kibana/issues/206710
This PR introduces the `InferenceChatModel` class, which is a langchain
chatModel utilizing the inference APIs (`chatComplete`) under the hood.
Creating instances of `InferenceChatModel` can either be done by
manually importing the class from the new `@kbn/inference-langchain`
package, or by using the new `createChatModel` API exposes from the
inference plugin's start contract.
The main upside of using this chatModel is that the unification and
normalization layers are already being taken care of by the inference
plugin, making sure that the underlying models are being used with the
exact same capabilities. More details on the upsides and reasoning in
the associated issue.
### Usage
Usage is very straightforward
```ts
const chatModel = await inferenceStart.getChatModel({
request,
connectorId: myInferenceConnectorId,
chatModelOptions: {
temperature: 0.2,
},
});
// just use it as another langchain chatModel, e.g.
const response = await chatModel.stream('What is Kibana?');
for await (const chunk of response) {
// do something with the chunk
}
```
### Important
This PR is only adding the implementation, and not wiring it anywhere or
using it in any existing code. This is meant to be done in a later
stage. Merging that implementation first will allow to have distinct PRs
for the integration with search (playground) and security (assistant +
other workflows), with proper testing
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Fixes https://github.com/elastic/kibana/issues/208958
## Summary
In this modal, we show a warning banner if there is no policies. The
problem was that the modal was render before the policies are loading.
This PR adds a `isLoading` state that ensures that this warning only is
shown if the policy list is empty after it has been fully loaded.
### How to test
1. Navigate to Index Management.
2. Create an index.
3. Select the index --> Click "Manage index" --> Click "Add lifecycle
policy"
4. Verify that no warning flashes before the policy shows.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Closes https://github.com/elastic/kibana/issues/209183
This PR re-enables the suites previously disabled on Defend Workflows
cypress tests.
The that led us to skip those suites are now resolved.
## Summary
Fixes https://github.com/elastic/kibana/issues/208915 !!
Fix monitor status rule for empty kql query results !!
1. Made sure if kql filter return no configs ids, rule break early to
not cover all monitors
### Testing
Create a synthetics rule with a kql filter which matches no monitors and
make sure rule doesn't trigger for other down monitors in the system
<img width="661" alt="image"
src="https://github.com/user-attachments/assets/ed0b3a1f-e8d1-4e22-a77d-1237ce557ac5"
/>
### Before
Create a rule and you can observe that rule would get triggered for all
monitors down in the system with matching condition criteria
## Summary
In https://github.com/elastic/kibana/pull/207739 we remove the
`includeComments` query param. This PR also removes the `comments` from
the response as part of the same effort.
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
Remove the implicit grant of the `savedQueryManagement` feature with the
Security Solution basic feature (ID: `siemV2`) in Serverless.
This is a follow-up of https://github.com/elastic/kibana/pull/202863
### Feature `siemV2`
This change only affects new roles created with the `siemV2` feature,
introduced recently
[here](https://github.com/elastic/kibana/pull/201780).
This change will align the Roles UI in Serverless and ESS, both
requiring the `savedQueryManagement` feature to be explicitly granted to
be able to manage saved queries.
### Feature `siem`
Roles using the deprecated `siem` feature will still implicitly receive
the `savedQueryManagement` feature (via an implicit grant of `discover`,
`dashboard`, `visualize`, and `maps`) + migration to their `*v2`
features which include `savedQueryManagement`. So there's no behavior
change for existing roles using the old `siem` feature (no breaking
change).
## Screenshots
The siem/siemV2 feature toggle:
<img width="774" alt="siem feature"
src="https://github.com/user-attachments/assets/2759988a-3cf8-4e1f-9431-16c09cf9d95c"
/>
The savedQueryManagement feature toggle:
<img width="774" alt="Saved query feature"
src="https://github.com/user-attachments/assets/d0145244-f4b8-4577-b91f-93f4dd1f758b"
/>
This PR adds Cypress test coverage for the Defend Insights component and
enables RBAC and tier validation tests. It should be merged after the
feature flag is enabled - https://github.com/elastic/kibana/pull/204242
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
All tests in
`deployment_agnostic/security_and_spaces/stateful.config_basic.ts` and
`deployment_agnostic/security_and_spaces/stateful.copy_to_space.config_basic.ts`
are intended to be run only with `basic` license, since FIPS overrides
it we need to skip that test for FIPS.
Separated index entries for `basic` and `trial` license, so tests with
trial config would still run.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Implements:
https://github.com/elastic/kibana/pull/207066#issuecomment-2609651683
I'm not sure how "smart" we need this to be. The `<EuiProgress />`
component already shows when either of these requests are running, but
the table doesn't auto-scroll back to the top when items are reloaded
(for example).
---------
Co-authored-by: Joe Reuter <johannes.reuter@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
