## Summary
Filters out any deprecated Kibana API usages from blocking upgrade
status.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### Risks
Should be mitigated by E2E tests
Fixes#200772
## 🐉 Summary
This PR fixes supporting data view runtime fields during rule execution
for the custom threshold rule.
## 🧪 How to test
1. Create a runtime field as shown below:
|Runtime field| Preview|
|---|---|
||
2. Make sure alerts are generated as expected both for regular and
no-data alerts:
### TODO
- [x] Add an API integration test
- [x] Test on MKI
## Summary
In https://github.com/elastic/kibana/pull/207739 we remove the
`includeComments` query param. This PR also removes the `comments` from
the response as part of the same effort.
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
This PR adds Cypress test coverage for the Defend Insights component and
enables RBAC and tier validation tests. It should be merged after the
feature flag is enabled - https://github.com/elastic/kibana/pull/204242
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
All tests in
`deployment_agnostic/security_and_spaces/stateful.config_basic.ts` and
`deployment_agnostic/security_and_spaces/stateful.copy_to_space.config_basic.ts`
are intended to be run only with `basic` license, since FIPS overrides
it we need to skip that test for FIPS.
Separated index entries for `basic` and `trial` license, so tests with
trial config would still run.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
This PR removes `Setup Guides` buttons.
Closes: #207026
> [!NOTE]
All the tests in
`src/platform/plugins/shared/guided_onboarding/public/components/guide_panel.test.tsx`
depend on those buttons so it was easier to remove those tests.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Users should use Lens to visualize data. Users should not use TSVB or
aggregation based panels. To this end, "easy button" UIs, such as
dashboard's "Add panel" flyout, should guide users to lens and not guide
them to legacy applications.
This PR removes `legacy` section from dashboard "Add panel" flyout.
Users can still add legacy panels to dashboards via "visualize"
application. The path is more steps, but this is as intended. Using old
stuff should be painful and require extra work as a caret to move users
to new stuff.
<img width="500" alt="Screenshot 2025-01-23 at 1 08 41 PM"
src="https://github.com/user-attachments/assets/e142cba9-a880-4ad6-a3f8-a981a03daaec"
/>
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This PR gets rid of the usage of the legacy "readdefinition" types in
the frontend except for the enrichment view, because @tonyghiani is
working on that in parallel.
I first looked into whether we can just use the existing
`IngestStreamGetResponse` type, but since it doesn't carry the name of
the stream, that is actually super annoying to use - passing the name as
a separate argument is pretty annoying and increases the risk of bugs.
Even though it's not fully "Elasticsearch"-y, I decided to add the
stream name to the `IngestStreamGetResponse` - this way the `stream`
property in the response is a regular `StreamDefinition`, so all the
functions that act on it can use it right away.
This simplifies the frontend code without hurting programmatic API
users.
Fixes https://github.com/elastic/streams-program/issues/103
This PR adds a constant keyword `stream.name` field that functions
similar to the `data_stream.dataset` field.
It will allow us to clearly associate a document or a set of documents
with their streams and filter data per stream efficiently. It's also
used to validate that documents are sent properly to the root stream
instead of targeting specific child streams directly.
The `stream.name` field reports as `keyword` in the UI, but is actually
mapped per index template as a constant keyword set to the respective
value.
## Summary
In 8.17 we have introduced `semantic_text`
https://github.com/elastic/kibana/pull/197007 which required dedicated
inference endpoint.
As we now have default `.elser-2-elasticsearch` inference endpoint
available we want to migrate it out, but it's not possible to just
override `inference_id` mapping for the Knowledge Base data stream, so
instead we decided to first update the mapping by adding
`search_inference_id` pointing to the `.elser-2-elasticsearch` (to make
sure the data is queryable without the dedicated endpoint). Then we
update the Data Stream mapping to use the default endpoint and after
that we rollover the DS index to make sure new index is created and new
inference endpoint is used for new Knowledge Base data ingestion.
Will add testing steps soon
## Summary
This PR reworks saved query privileges to rely solely on a single global
`savedQueryManagement` privilege, and eliminates app-specific overrides.
This change simplifies the security model for users, fixes bugginess in
the saved query management UI, and reduces code complexity associated
with maintaining two separate security mechanisms (app-specific
overrides and global saved query management privileges).
### Background
Saved queries allow users to store a combination of KQL or Lucene
queries, filters, and time filters to use across various applications in
Kibana. Access to saved query saved objects are currently granted by the
following feature privileges:
```json
[
"feature_discover.all",
"feature_dashboard.all",
"feature_savedQueryManagement.all",
"feature_maps.all",
"feature_savedObjectsManagement.all",
"feature_visualize.all"
]
```
There is also a saved query management UI within the Unified Search bar
shared by applications across Kibana:
<img
src="https://github.com/user-attachments/assets/e4a7539b-3dd4-4d47-9ff8-205281ef50e3"
width="500" />
The way access to this UI is managed in Kibana is currently confusing
and buggy:
- If a user has `feature_discover.all` and `feature_dashboard.all` they
will be able to load and save queries in Discover and Dashboard.
- If a user has `feature_discover.all` and `feature_dashboard.read` they
will be able to load queries in both Discover and Dashboard, but only
save queries in Discover (even though they have write access to the SO,
and API access). Instead they have to navigate to Discover to save a
query before navigating back to Dashboard to load it, making for a
confusing and frustrating UX.
- Access to the UI is even more confusing in apps not listed in the
above feature privileges (e.g. alerting, SLOs). Some of them chose to
check one of the above feature privileges, meaning users who otherwise
should have saved query access won't see the management UI if they don't
also have the exact feature privilege being checked. Other apps just
always show the management UI, leading to bugs and failures when users
without one of the above feature privileges attempt to save queries.
### Existing improvements
In v8.11.0, we introduced a new ["Saved Query
Management"](https://github.com/elastic/kibana/pull/166937) privilege,
allowing users to access saved queries across all of Kibana with a
single global privilege:
<img
src="https://github.com/user-attachments/assets/ccbe79a4-bd0b-4ed6-89c9-117cc1f99ee2"
width="600" />
When this privilege is added to a role, it solves the
`feature_discover.all` and `feature_dashboard.read` issue mentioned
above. However, it does not fix any of the mentioned issues for roles
without the new privilege. We have so far postponed further improvements
to avoid a breaking change.
### Approach
To fully resolve these issues and migrate to a single global privilege,
these changes have been made:
- Remove saved query SO access from all application feature privileges
and instead only allow access through the global saved query management
privilege.
- Stop relying on application feature privileges for toggling the saved
query management UI, and instead rely on the global privilege.
To implement this with minimal breaking changes, we've used the Kibana
privilege migration framework. This allows us to seamlessly migrate
existing roles containing feature privileges that currently provide
access to saved queries, ensuring they are assigned the global saved
query management privilege on upgrade.
As a result, we had to deprecate the following feature privileges,
replacing them with V2 privileges without saved query SO access:
```json
[
"feature_discover.all",
"feature_dashboard.all",
"feature_maps.all",
"feature_visualize.all"
]
```
Each area of code that currently relies on any of these feature
privileges had to be updated to instead access `feature_X_V2` instead
(as well as future code).
This PR still introduces a minor breaking change, since users who have
`feature_discover.all` and `feature_dashboard.read` are now able to save
queries in Dashboard after upgrade, but we believe this is a better UX
(and likely the expected one) and worth a small breaking change.
### Testing
- All existing privileges should continue to work as they do now,
including deprecated V1 feature privileges and customized serverless
privileges. There should be no changes for existing user roles apart
from the minor breaking change outlined above.
- Check that code changes in your area don't introduce breaking changes
to existing behaviour. Many of the changes are just updating client UI
capabilities code from `feature.privilege` to `feature_v2.privilege`,
which is backward compatible.
- The `savedQueryManagement` feature should now globally control access
to saved query management in Unified Search for all new user roles.
Regardless of privileges for Discover, Dashboard, Maps, or Visualize,
new user roles should follow this behaviour:
- If `savedQueryManagement` is `none`, the user cannot see or access the
saved query management UI or APIs.
- If `savedQueryManagement` is `read`, the user can load queries from
the UI and access read APIs, but cannot save queries from the UI or make
changes to queries through APIs.
- If `savedQueryManagement` is `all`, the user can both load and save
queries from the UI and through APIs.
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [x] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
This PR risks introducing unintended breaking changes to user privileges
related to saved queries if the deprecated features have not been
properly migrated, and users could gain or lose access to saved query
management on upgrade. This would be bad if it happened, but not overly
severe since it wouldn't grant them access to any ES data they couldn't
previously access (only query saved objects). We have automated testing
in place to help ensure features have been migrated correctly, but the
scope of these changes are broad and touch many places in the codebase.
Additionally, the UI capabilities types are not very strict, and are
referenced with string paths in many places, which makes changing them
riskier than changing strictly typed code. A combination of regex
searches and temporarily modifying the `Capabilities` type to cause type
errors for deprecated privileges was used to identify references in
code. Reviewers should consider if there are any other ways that UI
capabilities can be referenced which were not addressed in this PR.
Our automated tests already help mitigate the risk, but it's important
that code owners thoroughly review the changes in their area and
consider if they could have unintended consequences. The Platform
Security team should also review this PR thoroughly, especially since
some changes were made to platform code around privilege handling. The
Data Discovery team will also manually test the behaviour when upgrading
existing user roles with deprecated feature privileges as part of 9.0
upgrade testing.
---------
Co-authored-by: Matthias Wilhelm <matthias.wilhelm@elastic.co>
Co-authored-by: Matthias Wilhelm <ankertal@gmail.com>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: “jeramysoucy” <jeramy.soucy@elastic.co>
## Summary
This PR adds the following parameters to the `INIT` engine API:
* `frequency`: the transform run frequency
* `timeout`: the timeout for the initial creation of the transform
* `docsPerSecond`: transform throttling option. See
[here](https://arc.net/l/quote/vxcmfnhh)
* `delay`: The transform delay duration. See
[here](https://arc.net/l/quote/mzvaexhv)
Coming soon
In addition, the PR adds these fields to the Saved Object with the
engine descriptor, as well as providing a migration with the appropriate
backfilling.
Finally, there are some utility function that were/are helpful in
working with objects.
## How to test
*NOTE*: Always make sure the security default data view exists. Easiest
way it to just navigate to some Security UI.
### Checking the new defaults
1. Initialize an engine via dev tools by calling: `POST
kbn:/api/entity_store/engines/<entity_type>/init {}`
2. Call `GET kbn:/api/entity_store/status`. This response should now
contain all the default optional values.
### Observing the parameters are being applied
1. Initialize an engine via the API. This time pass any of the `timeout,
frequency, delay and docsPerSecond` options in the request body.
2. Once the `status` changes to `started`, query the respective
transform: `GET
_transform/entities-v1-latest-security_<entity_type>_default`
3. Check that the parameters have been applied to the transform
### Checking Saved Object Migration
1. Check out `main`.
2. Initialize the store.
3. Query `GET kbn:/api/entity_store/status`. Note down the fields in the
engine object.
4. Check out this branch.
5. Restart kibana.
6. Query `GET kbn:/api/entity_store/status` again. Observe the new
fields have been added and backfilled
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
## Summary
All tests in
`x-pack/test/spaces_api_integration/deployment_agnostic/security_and_spaces/stateful.config_basic.ts`
are intended to be run only with `basic` license, since FIPS overrides
it we need to skip that test for FIPS.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
Collects two events of graph visualization usage
1. When graph preview is shown - allows to determine how many users
ingested data that is graph compliant
2. When graph investigation component is visible - allows to determine
how many users entered the investigation user flow
**How to test:**
- Enable the feature flag
`kibana.dev.yml`:
```yaml
uiSettings.overrides.securitySolution:enableVisualizationsInFlyout: true
uiSettings.overrides.securitySolution:enableGraphVisualization: true
```
- Load mocked data:
```bash
node scripts/es_archiver load x-pack/test/cloud_security_posture_functional/es_archives/logs_gcp_audit \
--es-url http://elastic:changeme@localhost:9200 \
--kibana-url http://elastic:changeme@localhost:5601
node scripts/es_archiver load x-pack/test/cloud_security_posture_functional/es_archives/security_alerts \
--es-url http://elastic:changeme@localhost:9200 \
--kibana-url http://elastic:changeme@localhost:5601
```
- Make sure you include data from Oct 13 2024. (in the video I use Last
year)
- Run in dev tools:
```
POST kbn:/internal/telemetry/clusters/_stats?pretty=true&apiVersion=2
{
"unencrypted": true,
"refreshCache": true
}
```
- Check if the ui_counters were reported
**Resolves:** https://github.com/elastic/security-team/issues/10410
## Summary
We want to make Rule Customization available at higher license tiers.
### **Intended Workflows/UX**
#### **Basic/Platinum/Security Essentials License Tiers**
- **Editing Prebuilt Rules:**
- Allow the 8.16 behavior: only actions, exceptions, snoozing, and
enable/disable options can be modified.
- On the rule editing page, all tabs except *Actions* are disabled.
Disabled tabs will display a hover explanation:
- "Upgrade to Enterprise to enable prebuilt rule customization" for ECH.
- "Upgrade to Security Complete to enable prebuilt rule customization"
for Serverless.
<img width="356" alt="image"
src="https://github.com/user-attachments/assets/72e60933-aaaf-45a0-9660-4cd066d3afec"
/>
- Rule editing via API is not restricted (tracked separately:
https://github.com/elastic/security-team/issues/11504.
- **Bulk Actions:**
- Modifications to rule content via bulk actions are not allowed.
Prebuilt rules are excluded from bulk actions if the license level is
insufficient. Users will see an explanation for the exclusion.
- Serverless
<img width="737" alt="image"
src="https://github.com/user-attachments/assets/99fef72f-dd38-4c73-a9e3-7b4c8018b4ed"
/>
- ECH
- On the API level (`_bulk_action`), an error is returned if a user
tries to modify a prebuilt rule without the required license. Response
in this case looks like this:
```json
{
"statusCode": 500,
"error": "Internal Server Error",
"message": "Bulk edit failed",
"attributes": {
"errors": [
{
"message": "Elastic rule can't be edited",
"status_code": 500,
"rules": []
}
]
}
}
```
- **Rule Updates:**
- Updates are restricted to Elastic’s incoming updates only.
- The rule upgrade flyout is in read-only mode.
<img width="949" alt="image"
src="https://github.com/user-attachments/assets/16a56430-63e6-4096-8ffd-b97f828abdd4"
/>
- For previously customized rules where customization is now disabled
due to insufficient licensing, a notification will appear on the upgrade
flyout, clarifying that only an upgrade to Elastic's version is
available.
- On the API level (`_perform`), only requests with `pick_version =
target` are permitted. Requests with `rule.fields` values are not
allowed.
API response when `pick_version` is not `target`:
```json
{
"message": "Only the 'TARGET' version can be selected for a rule update;
received: 'CURRENT'",
"status_code": 400
}
```
API response when the `fields` value is provided:
```json
{
"message": "Rule field customization is not allowed. Received fields:
name, description",
"status_code": 400
}
```
- **Customized Rules:**
- Existing customizations remain intact, and the “Modified” badge is
retained
- On the rule management, monitoring, and update tables:
- On the rule update flyout:
- On the rule details page:
- When we edit a rule with customizations (e.g., change rule's actions),
the rule should stay marked as customized
- **Import/Export Scenarios:**
- These are handled separately
(https://github.com/elastic/security-team/issues/11502)
#### **Enterprise/Security Complete License Tiers**
- All rules can be fully edited
- Upgraded prebuilt or customized rules will have an editable view,
enabling full customization
Allow inherit/dsl lifecycles on unwired streams:
- inherit is a noop. it keeps the data streams lifecycle configuration
untouched
- dsl will apply a retention if the data stream is either unmanaged or
already using a dsl
Since we're still discussing how to apply ilm/fields changes on unwired
streams, ilm is currently disabled.
### Testing
- update an unmanaged/ilm/dsl unwired stream
```
PUT kbn:/api/streams/logs-foo-default
{
"stream": {
"ingest": {
"lifecycle": {
"dsl": {}
},
"routing": [],
"processing": [],
"unwired": {}
}
},
"dashboards": []
}
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
# Background
This change introduces a new Kibana task within the Security solution.
When the Security solution's entity store feature is enabled, the task
is scheduled in order to continuously reflect changes to the Security
solution's default data view, thereby updating the Transform associated
with the Entity Store when necessary.
# Implementation notes
A key problem when updating/upgrading a transform in the background is
that Elasticsearch requires a user to make the request for these
changes, but no "user" is present in background tasks. The internal
Kibana user does not suffice, because it does not always have access to
the underlying indices.
To accomplish the above, this PR leverages the Entity Manager's ability
to store the API Key of the user who installed the entity store, and
makes any associated changes to the Transform using that user's stored
API key. Said API key is encrypted, and uses a deterministic ID per
installed space in order to support later retrieval.
A single API key is installed per space, meaning multiple entity
"engines" in a space will leverage a single API key for updates.
# Steps to test locally
1. Pull down the code
2. To assist with a quicker feedback loop, manually edit the task's
`interval` constant to a low value, such as `1m`. This value can be
found
[here](x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/entity_store/tasks/data_view_refresh/constants.ts)
3. Start Kibana
4. Load desired entity test event data. (For example, the internal
[security-documents-generator](https://github.com/elastic/security-documents-generator)
repository's `yarn start entity-store` command can be used)
5. Enable the Security Entity Store by navigating to "Management->Entity
Store"
6. Validate the list of indices in the current transform by navigating
to `/app/management/data/transform` in Kibana, and clicking
"entities-v1-latest-security_host_default->JSON", looking for the
`source.index` field
7. Update the security default data view's index patterns to include a
new pattern. To do so, navigate to `/app/management/kibana/dataViews` in
Kibana, click the data view with the "Security Data View" badge, click
edit, and change the "Index pattern" by adding a comma and a new pattern
to the end, such as `,some-cool-pattern-*`. Save the change.
8. Wait the appropriate amount of time, as defined in step 2.
(Optionally check the Kibana console logs for activity.)
9. Once again, validate the list of indices within the transform as in
step 6, but this time see that the new index pattern is included.
---------
Co-authored-by: Pablo Machado <pablo.nevesmachado@elastic.co>
## Summary
This PR enables the SIEM migrations experimental flag by default in
ESS/on-prem.
We keep the experimental flag disabled in serverless (using
`config/serverless.security.yml`) since we don't want to release it yet.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Closes https://github.com/elastic/kibana/issues/186266
## Release notes
Enhanced Role management to manage larger number of roles by adding
server side filtering, pagination and querying.
## Summary
- Replaced the usage of Get Roles API with Query Role API
- Added server side pagination and filtering with a maximum limit of
10000 keys (default for max results on index). Added new label to
indicate that we show only 10k results.
- Search box replicates client side implementation by only filtering on
Role names.
### Run locally
Start ES ~with the JVM option to enable this feature~:
```
yarn es snapshot --license=trial
```
Start Kibana normally
```
yarn start --no-base-path
```
Navigate to Stack Management > Roles and verify the same behavior as the
screen recording below
### Screen recording
https://github.com/user-attachments/assets/a447e7df-8aa1-4044-a6b2-0aafe56844a9
## Technical notes
- Client side EuiInMemory table has been replaced by EuiSearchBar,
EuiBasicTable and Filters
- One new Kibana endpoint added
- `roles/_query`
- Replicates existing get_role endpoint by being public and added to
Open API spec
- Extra logic to handle previously UI only filter to show/hide reserved
roles
- Parse the query to construct the correct DSL if the filter is present
- Update Get All Roles by Space internal API to use the Query Role and
filter by space id using query DSL.
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This PR enables the feature flag for workflow insights, activates defend
insights within the Assistant scope, and aligns the tests accordingly.
The added privileges originate from [this
PR](https://github.com/elastic/kibana/pull/205088).
