Closes#163282
## Summary
This PR:
* Adds a `featureFlags.metricsExplorerEnabled` property to the Infra
plugin config to enable and disable Metrics Explorer depending on the
offering type
* Prevents `MetricsExplorerViewsService` initialization for serveless
based on the feature flag
* Prevents creating Metrics Explorer frontend routes when in serverless
* Prevents registration of the MetricsExplorerViews saved object when in
serverless
* Prevents initialization of the `metrics_explorer_views` API routes
when in serverless
**Trying to access Metrics Explorer in serverless**
<img width="1829" alt="CleanShot 2023-09-22 at 12 59 35@2x"
src="2b039925-0f0b-4c07-be29-bbe910de7a34">
**Trying to access views API**
<img width="1829" alt="CleanShot 2023-09-22 at 13 00 00@2x"
src="15269ec2-becd-4ee3-9b5e-d916df28a7b8">
**`infra/metrics_explorer` API still works as per ticket requirements**
<img width="1829" alt="CleanShot 2023-09-22 at 13 00 06@2x"
src="fb23f912-c6fd-46c8-9084-c17c51e5b064">
## How to test
* Checkout locally
* Enable Infra in `serverless.oblt.yml`: `xpack.infra.enabled: true`
* Run Kibana in serverless mode
* Try accessing `/app/metrics/explorer` route and make sure it's not
available
* Make sure other Infra routes (`/app/metrics/inventory` and
`/app/metrics/hosts`) still load as expected
* In Kibana dev console make sure you get 404 for `GET
kbn:/api/infra/metrics_explorer_views`
* Also check that you don't see `metrics-explorer-view` saved object in
the response for `GET
kbn:/api/kibana/management/saved_objects/_allowed_types`
* Run Kibana in non-serverless mode and make sure Metrics Explorer is
accessible and works as usual
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/167387
Replaced using kibana version when deciding if agent upgrade is
available (only in serverless, in stateful kibana version is still
returned as an available version).
To verify locally:
- [to test stateless] add this to `kibana.dev.yml`:
`xpack.fleet.internal.onlyAllowAgentUpgradeToKnownVersions: true`
- extract the `agent_versions_list.json` to local kibana folder
`~/kibana/x-pack/plugins/fleet/target`
[agent_versions_list.json.zip](12739519/agent_versions_list.json.zip)
- verify that upgrade available warnings still work if agent is < latest
agent version (8.10.2)
- when trying to upgrade agent, verify that the default version is the
latest agent version, and 8.11 is not in the list
Agent list:
<img width="1475" alt="image"
src="f06b7bc8-97e6-4ff9-b872-736ede5e969a">
Upgrade available filter - 1 agent on latest version, 9 upgradeable:
<img width="1314" alt="image"
src="4ff5ac02-903b-493b-94df-68b1b7ad6846">
Agent details:
<img width="1512" alt="image"
src="3ff6e1d5-2ccc-4814-83e5-c4760ad63722">
Agent on latest version has disable `Upgrade agent` action:
<img width="1322" alt="image"
src="f461dbf5-04e5-4bcc-8801-48c2b1a90225">
Bulk action with one agent that is not upgradeable (already on latest
version), expected error:
<img width="1597" alt="image"
src="8bfa46ae-6684-4748-9fca-e908c142b642">
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
I decided to remove the backticks suggestion as it was a copy paste from
SQL and is not going to solve ES|QL related errors. It also creates
confusion for many users.
<img width="1677" alt="image"
src="fe2a4fcb-c3e0-4d87-8568-32e7525f70d5">
Improves the display of long descriptions of transforms in the Transform
management page and when editing the description in the transform wizard
or edit flyout.
Previously If there was a long description, the text would not be
wrapped in the table on the management page, and it would not be
possible to view the full text in the text input when editing. This PR
adds line wrapping for the description column, and uses a text area for
editing the text.
Part of https://github.com/elastic/kibana/issues/163147
## Summary
Fixes a bug in ES|QL mode where you:
- have a query of index pattern 1 and select some fields
- change the query by using a different index pattern
- the selected columns do not reset
**BUG**
**NOW**
### Checklist
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
Implement risk score engine telemetry
Here we use 2 types of telemetry:
Event base telemetry:
- Risk execution success. With parameters `scoresWritten`,
`taskCompletionTimeSeconds`, `isRunMoreThanInteval`
- Risk execution error
Usage telemetry:
- `unique_user_risk_score_total` and `unique_host_risk_score_total` -
Total amount from latest transform index for host and users
- `unique_user_risk_score_day` and `unique_host_risk_score_day` - Last
day amount from the latest transform index for host and users
- `all_host_risk_scores_total` and `all_user_risk_scores_total` - Total
amount from datastream for all risk executions for host and users
- `all_host_risk_scores_total_day` and `all_user_risk_scores_total_day`
- Last day amount from datastream for all risk executions for host and
users
- `all_risk_scores_index_size` and `unique_risk_scores_index_size` -
sizes of datastream of all risk scores and latest transform index
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Updates the exceptions flyout UI `match_any` operator to accept numerous
duplicate values that differ in case. Prior to this change, a user could
not add a field value of `foo` and `FOO` - the UI would display that the
value is a duplicate. We now will allow this as exceptions are case
sensitive and this is a necessary use case for the current exceptions
behavior.
Cypress tests and FTR tests are added.
## Summary
This is hopefully the last batch of typescript issues to be fixed,
related to https://github.com/elastic/kibana/pull/166813.
It's also re-enabling full typecheck, with this, we should be back in a
clean, typechecked main branch.
Blocked by #167428
---------
Co-authored-by: Brad White <Ikuni17@users.noreply.github.com>
Co-authored-by: Brad White <brad.white@elastic.co>
Co-authored-by: Thomas Watson <watson@elastic.co>
Co-authored-by: Patryk Kopyciński <contact@patrykkopycinski.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
The osquery-cypress package was referring to configurations from a
parent folder. This caused difficulties with having to exclude/include
the same files in the `tsconfig.json`s, leading to hard-to-resolve
typescript issues.
This PR moves the configs in, and cleans up redundancies and type
errors.
chore(osquery): move osquery/cypress configs to cypress folder, fix type
errors
---------
Co-authored-by: Patryk Kopyciński <contact@patrykkopycinski.com>
Co-authored-by: Brad White <Ikuni17@users.noreply.github.com>
Co-authored-by: Brad White <brad.white@elastic.co>
## Summary
This PR is for adding the GCP Organization option as well as updating
the Single option to include Project ID field. Still rough
Changes:
- Added GCP Organization Option
- Project ID field now exist on Google Cloud Shell Single option as well
as Organization Option
- Organization ID field added to the form when user chose account_type :
GCP Organization
- Project ID are now optional (previously users aren't able to save the
integration without filling in the Project ID)
- Removed Beta tag for CIS GCP
TODO:
- Make sure previous installation using previous wont break because of
the new fields and requirement (migration)
- More tests
- Clean up
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Closes#167075
## Summary
Adds a public asset client available in the `setup` lifecycle hook for
plugins that depend on this one. `getHosts` is the only method available
on this client for now.
TODO, before merge:
- [x] Add docs for the server client
- [x] Add docs for the public client
- [x] Remove REST docs from plugin docs, not needed
- [x] Add unit tests for public client
### Testing this PR
One way of testing this new client is to apply the attached
test-assets.patch file locally, adjust the date range in the getHosts
query that is added in the infra plugin, and then start Kibana and
navigate to the infra app. You should see print out in the browser
console.
[test-assets.patch](12718693/test-assets.patch)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR supercedes https://github.com/elastic/kibana/pull/167107
Elastic Defend for Windows now collects ETW Threat Intelligence (ETW-TI)
events. Defend calls these API events. API events currently include the
existing Credential Access and ETW-TI. We will add more events under the
API umbrella in the future.
The Windows Events Policy `Credential Access` category has been renamed
to `API` in the UI and documentation - but it remains as
`credential_access` in the yaml for backwards compatibility. This new
category definition is a superset of the previous category.
Two new advanced options are added -
* `windows.advanced.events.api_disabled` - comma separated list
* `windows.advanced.events.api_verbose` - boolean
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
Resolves https://github.com/elastic/kibana/issues/165947
Resolves https://github.com/elastic/actionable-observability/issues/124
### Summary
This PR adds an Embeddable SLO Overview Widget to the Dashboard app. It
uses a [Metric
chart](https://elastic.github.io/elastic-charts/?path=/story/metric-alpha--basic)
component and displays an overview of the SLO health:
- name
- current sli value
- target
- status (background color)
### ✔️ Acceptance criteria
- The SLO widget should display the basic information listed above
- The SLO widget should be clickable and lead to the slo detail page
- The user should be able to select the SLO and filter to instanceId
- The tag "url.domain:mail.co" is the partition field and instanceId
value
<img width="1189" alt="Screenshot 2023-09-21 at 21 07 23"
src="03539b9d-23a5-45eb-aafb-df42e9421f77">
For more information regarding the key concepts and the usage of an
embeddable you can have a look at the Embeddable plugin
[README](https://github.com/elastic/kibana/tree/main/src/plugins/embeddable)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
### Error 1: Strict mode throws an error on non-filled variables. I
forgot to add this when creating the script
```
.buildkite/scripts/steps/es_serverless/promote_es_serverless_image.sh: line 40: UPLOAD_MANIFEST: unbound variable
```
### Error 2: Uploading multi-arch images
https://elastic.slack.com/archives/C5UDAFZQU/p1695725623585409
We've noticed that the downloaded `latest-verified` images are slower to
start up than normal. After inspecting the manifests, it seems we were
getting `linux/amd64` on our arm devices as well.
The solution is to grab and upload both platform variants. (using this
blog:
https://www.docker.com/blog/multi-arch-build-and-images-the-simple-way/)
This PR is next step completing discover timeline integration. All
previous/nest steps have been defined here:
https://github.com/elastic/security-team/issues/6677
## Summary
This PR implements the integration between timeline State v/s Discover
State. The purpose of this PR is to add functionality related to the
persistence of saved search which will always be linked to the timeline
user is working in.
Below diagram shows briefly how saved search is working with timeline.
```mermaid
graph TD;
DS(Discover State) -. user updates .-> SS(Saved Search);
SS(Saved Search) -. updates savedSearchId .-> TS(Timeline State) ;
TS(Timeline State) -. restores Saved Search to App state .->DS(Discover State);
```
Primarily, this PR implements below technical components:
1. `DiscoverInTimleineContext` : provides the ability across security
solution to manipulate discover state.
2. `useDiscoverInTimelineActions`: acts as a helper to provide
short-hand actions to manipulate discover state. For eg.
`resetDiscoverAppState` or `restoreAppStateFromSavedSearch`.
Here is the small demo video:
006465ba-19ce-4209-ac46-21dbb746508d
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Michael Olorunnisola <michael.olorunnisola@elastic.co>
## Summary
This PR follows #166460 by adding Category panels to the Form.
<img width="1807" alt="Screenshot 2023-09-27 at 3 36 16 PM"
src="2abe8cf5-5822-473f-affd-148fb7949316">
## Notes
This PR is divided into several commits, the first few being
prerequisite codemods. I recommend reviewing each commit separately, as
the codemods might obscure the actual component work.
- [e78586f - Make SettingType pre-defined to clean up
references](e78586fe44)
- This makes the `SettingType` optional, to clean up areas where the
generic need not be specific.
- [80a9988 - [codemod] Make onFieldChange and onInputChange more
distinct](80a9988516)
- The `onChange` handlers weren't very clear as you work your way up the
component tree. This makes the implementation and usage easier to
understand, (and easier to [replace with state
management](https://github.com/elastic/kibana/issues/166579)).
- [5d0beff - [fix] Fix logged errors in form
tests](5d0beff00c)
- This fixes some logged errors in the Form from `Monaco` and from some
missing `act` and `waitFor` calls.
Closes#161754Closes#166807
To make the testing and review easier I merged the old components
[cleanup PR](https://github.com/jennypavlova/kibana/pull/5) into this
one
## Summary
This PR replaces the old node details view with the asset details flyout
### Old
### New
### Testing
1. Go to inventory
2. Click on a host in the waffle map
3. Click on any **host**
- These changes are related only if a `Host` is selected- in the case of
a pod the view shouldn't be changed:
4. Check the new flyout functionality
3557821c-7964-466e-8514-84c2f81bc2fd
Note: the selected host should have a border like in the previous
version (this I fixed in the [last
commit](ff4753aa06))
so it should be added if there is a selected node:
<img width="1193" alt="image"
src="6646fe47-6333-435a-a5ec-248339402224">
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Correct the `license_uuid` field name in the Endpoint Policy. Before, it
was named `license_uid`, but the Endpoint expects `license_uuid`.
This PR in intended to be backported to `8.10.3` which brings up an
interesting problem since we already have a migration added to `main`
for the `8.11` release.
After talking with the kibana-core team, my approach is to add the
migration for this bug fix to this PR. Then, to keep the `modelVersions`
consistent, I will backport all `modelVersions` to `8.10.3` to keep the
migrations consistent. Keeping these consistent is important so that
both users upgrading from `8.10.x` in ESS and the Serverless line all
remain in sync. The end result is that the policies inside of of
`8.10.3` will have an extra field that will be unused until `8.11.0`
The following `8.10.3` backport for this will include the extra
migration and I will request reviews for it since it will be more than a
normal backport.
Policy:
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
* Adds API versioning to all routes involved in new Risk Engine, public
and private
* Adds missing PLI auth headers for some routes
* Updates API invocations to specify an appropriate version header
* Does NOT add header to legacy transform-based EA routes
### Checklist
- [x] Verify no API calls from the UI were missed
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
- Adds the missing `system_indices_superuser` role to the `role.yml`
file used when starting ES for serverless and using the security
solution override files for resources
## Summary
This PR adds Gated form when user visit Workplace Search , when`
kibana_uis_enabled == false`. User will not be able to able to access
any other Workplace search routes other than Overview page.
**Note**: Form submission and API call be included in next PR
Screen Recordings :
672a7b2e-3e5f-4fa1-8535-b5080b3a2dfc
7c8129cf-6f50-4039-9b50-b9a655361bd1
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Updated EMS Styles are waiting to be put into production. They are
already available in Elastic staging environment
([preview](maps-staging.elastic.co/?manifest=testing)). This PR is a
safe measure to ensure that this change do not break our CI tests.
The process has been as follows:
1. Momentarily replaces the EMS Tile Service `tileApiUrl` by our staging
server to force the use of the new styles and check which tests break
with the slightly different basemaps at
[12481c6](12481c6ada)
2. Look for related [broken
tests](https://buildkite.com/elastic/kibana-pull-request/builds/161870)
```
Error: expected 0.030813687704837327 to be below 0.03
```
4. Adjust the threshold for the dashboard report, since the new value
was slightly over the limit
[e655b84](e655b84569)
5. Wait for a green CI (this took a few days because of unrelated issues
with Kibana CI)
6. Revert the `tileApiUrl` change to its original value
[c0030bc](c0030bcff1)
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR adds the serverless FTR tests that we already have in the [QA
quality
gate](https://github.com/elastic/kibana/blob/main/.buildkite/pipelines/quality-gates/pipeline.tests-qa.yaml#L18-L24)
to the staging quality gate.
### Details
We intentionally decided run the same set of FTR tests again in staging
for starters. We're accepting the over-testing here until we have enough
confidence and experience with our serverless product stability to
decide which set of tests to run in which environment.
This PR also explicitly sets the `EC_ENV` and `EC_REGION` environment
variables for QA and Staging. It worked fine for QA env so far without
setting the environment variable because it fell back on the QAF default
values. Setting these values explicitly, makes it more robust.
## Summary
Adds Open API definition for `GET /api/fleet/uninstall_tokens`, which is
hidden behind feature flag for now, but **planned to be enabled for
v8.11.0**.
This should be merged with:
- https://github.com/elastic/kibana/pull/166794
## Summary
Close https://github.com/elastic/kibana/issues/167152
Log a warning instead of throwing an error in
`saved_object_content_storage` when response validation failed.
We decided to do this as a precaution and as a follow up to an issue
found in saved search https://github.com/elastic/kibana/pull/166886
where storage started failing because of too strict validation.
As of this PR the saved_object_content_storage covers and this change
cover:
- `search`
- `index_pattern`
- `dashboard`
- `lens`
- `maps`
For other types we agreed with @dej611 that instead of applying the same
change for other types (visualization, graph, annotation) the team would
look into migrating their types to also use
`saved_object_content_storage`
https://github.com/elastic/kibana/issues/167421
resolves: #158403
When conflicts are detected while updating alert docs after a rule runs,
we'll try to resolve the conflict by `mget()`'ing the alert documents
again, to get the updated OCC info `_seq_no` and `_primary_term`. We'll
also get the current versions of "ad-hoc" updated fields (which caused
the conflict), like workflow status, case assignments, etc. And then
attempt to update the alert doc again, with that info, which should get
it back up-to-date.
Note that the rule registry was not touched here. During this PR's
development, I added the retry support to it, but then my function tests
were failing because there were never any conflicts happening. Turns out
rule registry mget's the alerts before it updates them, to get the
latest values. So they won't need this fix.
It's also not clear to me if this can be exercised in serverless, since
it requires the use of an alerting framework based AaD implementation
AND the ability to ad-hoc update alerts. I think this can only be done
with Elasticsearch Query and Index Threshold, and only when used in
metrics scope, so it will show up in the metrics UX, which is where you
can add the alerts to the case.
## manual testing
It's hard! I've seen the conflict messages before, but it's quite
difficult to get them to go off whenever you want. The basic idea is to
get a rule that uses alerting framework AAD (not rule registry, which is
not affected the same way with conflicts (they mget alerts right before
updating them), set it to run on a `1s` interval, and probably also
configure TM to run a `1s` interval, via the following configs:
```
xpack.alerting.rules.minimumScheduleInterval.value: "1s"
xpack.task_manager.poll_interval: 1000
```
You want to get the rule to execute often and generate a lot of alerts,
and run for as long as possible. Then while it's running, add the
generated alerts to cases. Here's the EQ rule definition I used:
I selected the alerts from the o11y alerts page, since you can't add
alerts to cases from the stack page. Hmm. :-). Sort the alert list by
low-high duration, so the newest alerts will be at the top. Refresh,
select all the rules (set page to show 100), then add to case from the
`...` menu. If you force a conflict, you should see something like this
in the Kibana logs:
```
[ERROR] [plugins.alerting] Error writing alerts: 168 successful, 100 conflicts, 0 errors:
[INFO ] [plugins.alerting] Retrying bulk update of 100 conflicted alerts
[INFO ] [plugins.alerting] Retried bulk update of 100 conflicted alerts succeeded
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
