## Summary
Vastly improve performance of the Fleet final pipeline's date processing
for the `event.ingested` field by using a `script` processor and
`DateTimeFormatter`. All credit to @joegallo for the performance
improvements.
Closes https://github.com/elastic/kibana/issues/157430
Our test coverage for the `event.ingested` field
[here](090569bb45/x-pack/test/fleet_api_integration/apis/epm/final_pipeline.ts (L109-L127))
should suffice for ensuring we're not breaking anything here.
## Screenshots
I fired up a local cluster and ran Fleet Server + a single Agent to
verify that `event.ingested` is set correctly, and that other pipeline
fields like `agent.id` are working as expected.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
- https://github.com/elastic/kibana/pull/166781 PR added category id for
each Security rule type. This PR was created before ES|QL rule was
added, but ES|QL rule got merged earlier, before the last successful
build for that PR was created. That allowed maintenance window PR to be
merged, without adding category to ES|QL rule
## Summary
This PR follows https://github.com/elastic/kibana/pull/167466, after
deploying the GPCTL in production. After separating the logs from
nonprod and prod GPCTL, we have to go to 2 different dashboard based on
the environment we would like to check (non-prod/prod)
## Summary
PR https://github.com/elastic/kibana/pull/166924 introduces a bunch of
integration tests that are causing that specific group to take a lot
longer to execute:
The goal of this PR is to move these new tests to a separate group, so
that both groups take 20-25mins.
## Summary
Resolves: https://github.com/elastic/kibana/issues/166301
Adds support for solution/category filtering to maintenance windows by
adding a new property: `category_ids`. Selecting one or more solutions
when creating/updating a maintenance window will cause the maintenance
window to only suppress rule types belonging to said solutions. In order
to achieve filtering by solution/category, we are adding a new field to
the rule types schema called `category`. This field should map to the
feature category that the rule type belongs to (`observability`,
`securitySolution` or `management`).
Our initial plan was to use feature IDs or rule type IDs to accomplish
this filtering, we decided against using rule type IDs because if a new
rule type gets added, we need to change the API to support this new rule
type. We decided against feature IDs because it's a very anti-serverless
way of accomplishing this feature, as we don't want to expose feature
IDs to APIs. We decided on app categories because it works well with
serverless and should be much easier to maintain if new rule types are
added in the future.
This means the `rule_types` API has to be changed to include this new
field, although it shouldn't be a breaking change since we're just
adding a new field. No migrations are needed since rule types are in
memory and maintenance windows are backwards compatible.
### Error state:
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Dima Arnautov <arnautov.dima@gmail.com>
Closes#166136
## Summary
This PR removes the groups in the rules list in the observability.
|With disabled rule|Without disabled rule|
|---|---|
||
## 🧪 How to test
- Check the rule type list in observability > rules page
- Check the filter works as before
## Summary
Re-enable a subset of cypress serverless tests:
- Restructured
`x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_creation/custom_query_rule.cy.ts`
to break it down into and enable the following tests:
- `rule_creation/common_components.cy.ts`
- `rule_creation/custom_query_rule.cy.ts`
- `rule_details/common_components.cy.ts`
- `rule_details/custom_query_rule.cy.ts`
- `rule_edit/custom_query_rule.cy.ts`
## Summary
Close https://github.com/elastic/kibana/issues/159691
[Requirements](https://docs.google.com/document/d/1uXgyDIGuIqkYXmavdMTpBgQEiOP07ObJYb7GNq5GiSE/edit#heading=h.okl11rz12ytg)
[A/B test
description](https://docs.google.com/document/d/1yzfZF8mtlRNH4X6HjosD6Exh24zAhN__LR_zh3DOzKw/edit?usp=sharing)
[Figma](https://www.figma.com/file/WGhmfgyy9FBOltLtycfGPE/Getting-started?type=design&node-id=92-44804&mode=design&t=mwbIexn5Fs754HQz-0)
Testing - see _testing_ section below for more details
(https://dosant-pr-167374-d-2023-06-14-global-drift-with-experiment.kbndev.co/.
elastic/changeme)
This PR enables cloud chat (Drift) globally. This is done by adding a
custom chat button in the Kibana header which manually toggles Drift
widget. We attempt to manually position the widget to the top of the
screen so it pops up close to the chat button that triggered it.
Previously Drift chat was available only on specific pages like
Solutions onboarding pages, integrations, setup guides as a regular chat
widget with the floating chat bubble in the bottom right corner. We
couldn't enable it on all pages, because on a lot of them the floating
chat bottom would have overlapped the application UI.
We also were asked to add [an a/b
test](https://docs.google.com/document/d/1yzfZF8mtlRNH4X6HjosD6Exh24zAhN__LR_zh3DOzKw/edit?usp=sharing):
- A: The chat button appears in the header for all pages (new)
- B: The chat button appears as floating action button in the
bottom-right corner on pages where Drift was previously available
(solutions onboarding pages, integrations, setup guides)
### Screenshots / Videos
#### Global Chat in the header
#### The tour on the first appearance
#### (Part of A/B test) Drift in the header on new pages and as floating
action button on old pages
0386ccbd-ab6c-4eb2-a57b-f9324fcf73eb
### Implementation notes
- **We still enable Drift only for trial users + gap window**
- We exposed additional APIs from drift iframe to manually control its
visibility and react to more events
https://github.com/elastic/cloud/pull/118761. This changes are required
for the code in this PR to work. ~The updated frame code wasn't deployed
yet.~ the changes were deployed
- We use [`playbookFired` event
](https://devdocs.drift.com/docs/drift-events#playbook-fired) to know if
Drift chat should be visible for the current user. We show the button in
the header only when it fires.
- To react to the event and to display the button, we have to kick of
Drift iframe initialization first
- This means Drift codes loads before we show the button and before user
interacts with it (Only when Drift is enabled, meaning, only for trial
users + gap window)
- Subsequent launches or opens of the same playbook will not re-trigger
the `playbookFired` event, I used local storage flag to workaround this
and show the live chat button, but it also has it's own edge case. As an
alternative we can always show the chat button and don't rely on the
playbook event, more details here:
https://docs.google.com/document/d/1j313mVOIz19Rkoj8TDFWaLc7Pgk_ByBBKJFNIC-jbyc/edit?usp=sharing.
For now was decided to rely on the `playbookFired` event.
- A/B: **to support for both new and old implementation, I had to
refactor the old one from drop-in chat to global chat that is controlled
by list of hardcoded URLs.** This is not ideal, but this allows two
implementations to co-exist with much tech-debt and we plan to get rid
of this after the a/b test.
- When we navigate between pages with different implementations, Drift
re-initializes itself (just like in old implementation), this
performance debt should go away when we get rid of the a/b test.
- When end-to-end testing the a/b experiment, I found a bug in the a/b
test setup in Kibana https://github.com/elastic/kibana/issues/167240
this needs to be addressed separately for a/b test to work properly.
- When the user receives a message from Drift - the custom "Live Chat"
button doesn't indicate that there is a new message. This needs a follow
up, @Dosant to create an issue
### Testing
Version with the experiment where on old pages Drift appears as a
floating chat bubble -
https://dosant-pr-167374-d-2023-06-14-global-drift-with-experiment.kbndev.co/
elastic/changeme
> [!NOTE]
> If the live chat button doesn't appear, it is likely because the
playbook was recently activated by someone else. you can workaround this
for testing by creating a different user or by setting
`cloudChatPlaybookFiredOnce : true` to localstorage. This issue and
mitigation is described in details in the "implementation details"
section
#### To test locally:
```
xpack.cloud.id: 'some-id'
xpack.cloud.trial_end_date: '2023-09-21T00:00:00.000Z'
xpack.cloud_integrations.chat.trialBuffer: 45
xpack.cloud.chat.enabled: true
xpack.cloud.chatIdentitySecret: <pls react out>
xpack.cloud.chat.chatURL: https://elasticcloud-production-chat-us-east-1.s3.amazonaws.com/drift-iframe.html
xpack.cloud_integrations.experiments.flag_overrides:
"cloud-chat.chat-variant": "bubble" or "header"
```
## Summary
Bump Cypress-related dependencies to the latest versions and update
`renovate.json` to do it automatically in the future
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Gloria Hornero <gloria.hornero@elastic.co>
- Added new Profiling settings so users can customize the CO2 variables
- Fixed Embeddable components to also read the new settings
- Moved code from APM to obs-shared to create the custom settings page
in Profiling.
- New Settings Page was created in Profiling UI so users can easily find
the settings:
<img width="2053" alt="Screenshot 2023-09-22 at 11 18 35"
src="6969b079-745d-4302-8ff2-4f0f256c7f51">
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
A follow up for https://github.com/elastic/kibana/pull/165866
## Summary
This PR replaces icons for the sidebar toggle button.
<img width="200" alt="Screenshot 2023-09-29 at 13 00 39"
src="6ed10562-9a50-48ce-b6b2-030ab7b11e11">
<img width="200" alt="Screenshot 2023-09-29 at 13 00 47"
src="e286e152-a6bb-4b21-a97a-44419757dafb">
Moves the Dashboard view mode from session storage to local storage. This means that users will only need to enter edit mode **once** if they are an editor, and any subsequent Dashboards they open will already be in edit mode.
This fixes a problem where versioned APIs were not supported. It also
adds a `--local` flag for easily running the diagnostics tool against a
local cluster running with default credentials (elastic/changeme)
## Summary
Adds new advanced Policy options for Mac for 8.11
<img width="1728" alt="image"
src="0948817b-32b4-4573-90b1-756d41a38d42">
<img width="1728" alt="image"
src="297d7d51-7785-402a-918f-e41fc30cf211">
Enrichment fields:
<img width="1728" alt="image"
src="f8ddc1de-c118-4bd4-9c57-b88fec957b8f">
<img width="1726" alt="image"
src="04087c19-7533-41ab-938e-868a8570a3eb">
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Daniel Ferullo <56368752+ferullo@users.noreply.github.com>
Closes#161419
## Summary
Replaces deprecated EuiPage*_Deprecated components with the suggested
replacement components (see
[issue](https://github.com/elastic/kibana/issues/161419)).
### Visual Difference
The only UI that is noticeably affected is the User Profile page. The
spacing and header font size are slightly different from the previous
implementation, however, it is consistent with other
`Eui/KibanaPageTemplate` pages.
Previous render:
<img width="1391" alt="Screenshot 2023-09-22 at 1 27 27 PM"
src="58d2a86e-63b7-4c7d-bcf1-2071290d25b2">
New render:
<img width="1286" alt="Screenshot 2023-09-22 at 2 56 44 PM"
src="0f2b3e38-8ea3-4daf-8937-03e47474658c">
### Applicable Tests
-
x-pack/plugins/security/public/account_management/user_profile/user_profile.test.tsx
-
x-pack/plugins/security/public/management/users/users_grid/users_grid_page.test.tsx
-
x-pack/plugins/spaces/public/management/edit_space/manage_space_page.test.tsx
-
x-pack/plugins/spaces/public/management/spaces_grid/spaces_grid_page.test.tsx
-
x-pack/plugins/security/public/management/roles/roles_grid/roles_grid_page.test.tsx
(for permission denied)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Use legacy risk score indices if the new risks engine wasn't
installed
Legacy risk scores indices:
- `ml_user_risk_score_latest_default`
- `ml_host_risk_score_latest_default`
New risk score indices:
- `risk-score.risk-score-latest-default`
Which parts use risk score indices?
- Entity Analytics dashboard
- All host/user risk tab
- Single host/user risk tab
- Alert enrichment UI
- Alert enrichment Backend
All these components can use only one risk score data source at a time,
a new or legacy one.
Previously we started to use the new risk score indices if FF -
`['riskScoringPersistence', 'riskScoringRoutesEnabled'] ` was present
and there was no possibility for the client to see old risk score data
in UI, or use it for alert enrichments
This PR, changes the way, that clients will use legacy risk score
indices until they specifically choose to upgrade the risk engine.
In this video legacy risk transforms are installed, but user risk
transforms are not installed.
16acf38a-5694-4706-9426-cdea58b70a9b
### Cypress tests
I modify those tests in a way that we test that it works still with
legacy risk scores, but also with new risk scores.
Also, add some tests for upgrade workflow.
Also, some risk scores a different in legacy and new indices for the
same entities. So we can show in the tests, that they use different
indices.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
The Mobile product is moving from technical preview to beta. This change
reflects that in the UI.
Before:
<img width="659" alt="Screenshot 2023-09-25 at 4 15 13 PM"
src="59b85ca2-7feb-4899-bf6c-54da0c66fb51">
After:
<img width="692" alt="Screenshot 2023-09-25 at 4 02 50 PM"
src="7b52451b-0f0d-415e-8679-0f19c6e83094">
<img width="770" alt="Screenshot 2023-09-25 at 4 04 54 PM"
src="0898511a-77ab-4c3f-ac3a-ca22513f8593">
## Summary
This PR adds tracking for Log Rate Analysis and Log Pattern Analysis
endpoints for AIOps.
- tracks type of analysis and source (where the analysis is being run
from)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR adds a Settings application component for rendering the Advanced
Settings page in serverless.
### How to test:
1. Start Es with `yarn es serverless` and Kibana with `yarn
serverless-{es/oblt/security}`
2. Go to Management -> Advanced Settings
3. Verify that the settings can be changed and saved.
### Advanced Settings page:
<img width="1495" alt="Screenshot 2023-09-28 at 20 56 25"
src="374b3bbd-7bf6-4de7-8129-8b293dd1698e">
### Added an Advanced Settings card to the Management landing page:
<img width="1575" alt="Screenshot 2023-09-28 at 12 24 23"
src="c08b8b36-ff40-4772-87d6-597629d78342">
<!--- ### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
-->
---------
Co-authored-by: Clint Andrew Hall <clint@clintandrewhall.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Clint Andrew Hall <clint.hall@elastic.co>
## Summary
This PR introduces a new `internal/elastic_assistant/evaluate` route and
`Evaluation` Advanced Setting within the Assistant for benchmarking and
testing models, agents, and other aspects of the Assistant
configuration.
Enable via the `assistantModelEvaluation` experimental feature in your
`kibana.dev.yml` (and better add `discoverInTimeline` for good measure
as well! :)
> xpack.securitySolution.enableExperimental:
['assistantModelEvaluation', 'discoverInTimeline']
Then access from within the `Advanced Settings` modal in the Assistant.
To use, first select your Connectors/Models, then corresponding Agent
configurations, then what model you would like to use for final
evaluation, the evaluation type, and if `custom`, you can specify the
evaluation prompt that is sent off to the evaluator model. Finally,
specify the `dataset`, and `output index` that the results should be
written to, then click `Perform evaluation`.
Sample datasets can be found in
`x-pack/plugins/elastic_assistant/server/lib/model_evaluator/datasets`,
and include:
* `esql_dataset.json`
* `query_dataset.json`
* `security_labs.json`
* `security_questions_dataset.json`
<p align="center">
<img width="500"
src="99f8e764-34bc-4eb7-bbd8-7038ab72117b"
/>
</p>
<p align="center">
<img width="500"
src="f48f91dc-45da-4cd6-9dc7-cb88105668b2"
/>
</p>
### Checklist
Delete any items that are not applicable to this PR.
- [X] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
https://github.com/elastic/security-team/issues/7442
This PR:
1. Sets `protectionUpdatesEnabled` feature flag to `true`.
2. Text changes requested
[here](https://github.com/elastic/security-team/issues/7442)
3. Introduces ~~dismissable~~ callout that warns user about consequences
of turning auto updates off.
4. Adds cypress test that validates note persistance.
## Summary
This PR refactors the public plugin contract of Security Solution,
grouping exposed components into one component service to reduce the
boilerplate code needed when adding new components to the plugin
contract for serverless.
It also refactors the `isILMAvailable` flag that was exposed in the
_start_ contract by the `dataQualityPanelConfig`, which is more
self-explanatory, and it has been exposed in the _setup_ contract
without observable.
### Usage of `ContractComponentsService`
1. Define the component in ess or serverless plugins:
```
securitySolution.setComponents({
getStarted: getSecurityGetStartedComponent(services, productTypes),
dashboardsLandingCallout: getDashboardsLandingCallout(services),
// ... other components
});
```
2. Use the component in the main security plugin
```
const { services: { getComponent$ } } = useKibana();
const GetStartedComponent = useObservable(getComponent$('getStarted'));
return <>{GetStartedComponent}</>;
```
Component names are defined at
84583e4960/x-pack/plugins/security_solution/public/contract_components.ts (L11)
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Ref: https://github.com/elastic/kibana/issues/166971
Warning messages such as "max signals reached" or "rule may not have
read access to these indices" can now write metrics (if available) to
the rule execution logger.
Warning message:
<img width="1124" alt="warning_no_access_to_logs_index"
src="70b63a47-3e54-45b1-ba49-da531595e47e">
Metrics:
<img width="1669" alt="metrics_no_access_to_logs_index"
src="eac91dff-316a-4587-bab9-c816947be00e">
