Commit graph

10732 commits

Author SHA1 Message Date
Tiago Costa
87f6c1d4b0
skip flaky suite (#167914) 2024-06-08 00:09:45 +01:00
Kibana Machine
817842b73a skip failing test suite (#185024) 2024-06-08 06:13:16 +10:00
Ignacio Rivas
1e197cf718
[Ingest Pipelines] Add unsaved changes prompt (#183699) 2024-06-07 20:35:56 +02:00
Carlos Crespo
860f8dbf13
[Serverless][Observability] Use roles-based testing - api_integration (#184654)
part of: [#184033](https://github.com/elastic/kibana/issues/184033)

## Summary

This PR changes the observability serverless tests (and impacted
security tests) to not run with operator privileges.


### How to test

- Follow the steps from
https://github.com/elastic/kibana/blob/main/x-pack/test_serverless/README.md#run-tests-on-mki
2024-06-07 08:34:28 -07:00
Vitalii Dmyterko
93c45874a3
[Security Solution][Detection Engine] removes suppression terms from alert id (#184453)
## Summary

- Removes suppression terms from list of properties alert id is
generated.
- As part of
[discussion](https://github.com/elastic/kibana/pull/181926#discussion_r1593337828)
we decided that alerts generated from suppression in memory do not need
to have suppression terms as part of id generation. This would prevent
creating duplicate alerts for different suppression configurations
- flaky tests
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6171
2024-06-07 15:50:42 +01:00
Pablo Machado
90ae0f22e3
[SecuritySolution] Remove transform delay time + add unattended setting (#184797)
## Summary

* Refresh risk score index after persisting risk score
* Schedule the risk score transform after calculating the risk score for
a single entity
* Update transform config 
  * Add `delay: 0s`
  * Add `managed` and `managed_by` metadata 
  * Add `version` metadata (used by the migration)
  * Add `unattended: true`
* Create a transform migration


## How to test it?
### Migration
1. Install risk engine on an old version
2. Upgrade version
3. Add new alerts with new host and user
4. Run the risk engine (you can wait or force it to run by switching the
flag on/off)
5. Open the explore page (user|host)/risk-tab and check if the new
user|host is present

### New installation
1. Install risk engine on an empty cluster
3. Add new alerts with new host and user
4. Run the risk engine (you can wait or force it to run by switching the
flag on/off)
5. Open the explore page (user|host)/risk-tab and check if the new
user|host is present






### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
2024-06-07 05:34:01 -07:00
Ignacio Rivas
ab151751ff
[Ingest Pipelines] Add new two columns detail layout to pipeline details page (#181003) 2024-06-07 12:42:54 +02:00
Marco Antonio Ghiani
83021ed254
[Logs Explorer] Move table control actions column to the left (#184817)
## 📓 Summary

Closes #184735 

These changes move the control actions on the Logs Explorer data grid
rows to the left for more visual consistency with Discover.

<img width="2394" alt="Screenshot 2024-06-05 at 12 53 45"
src="80c36bce-739e-4dbd-89de-fda58095da89">

---------

Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
2024-06-07 11:12:24 +02:00
honeyn303
1ff87eb551
Gemini connector integration (#183668) 2024-06-06 11:52:35 -06:00
Cristina Amico
966736e4b4
[Fleet] Add rate limiting to install by upload endpoint (#184036)
Fixes https://github.com/elastic/ingest-dev/issues/3217

## Summary

Add rate limiting to "install by upload" endpoint. 
Implemented with a cache that is set with the timestamp of each install
by upload, independently from the package name/version. If the time
elapsed since the last timestamp it's less than retry time (10s), the
endpoint will return `429 Too many requests`.

### Testing
- Upload a package with 
```
curl -XPOST -H 'content-type: application/zip' -H 'kbn-xsrf: true' http://localhost:5601/YOUR_PATH/api/fleet/epm/packages -u elastic:changeme --data-binary @PACKAGE_NAME.zip
```
- Upload another package shortly after. It can be the same one or
another one, as the rate limiting is applied across all uploads, no
matter the package name.
- If the second upload happens <10s after the first one, should return
error `429 Too Many Requests. Please wait 10s before uploading again.`

### Checklist

- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2024-06-06 17:13:52 +02:00
Pablo Machado
b4561e7c3e
[Security Solution] Change risk scoring sum max and simplify risk score calculations (#184638)
## Summary

* Update the `RISK_SCORING_SUM_MAX` to the appropriate value based
10.000 alerts (read more on the original issue)
* The following risk scoring engine lines can be simplified by no longer
multiplying by 100, and instead using the value above directly. I also
renamed the constants to improve reliability,


I rounded `2.592375848672986` up to `2.5924` so the calculated score
won't go above `100`.

For `10.000` alerts with a risk score of `100` each the calculated risk
score is `99.99906837960884`

Risk score calculation for 10_00 alerts with 100 risk score
![Screenshot 2024-06-03 at 11 56
48](00c876ea-388b-4322-b8f8-19fc65f9f833)

Risk score calculation for 1_000 alerts with 100 risk score
![Screenshot 2024-06-03 at 11 57
29](929746c2-19e9-4da1-b4b1-c6e56edfc77c)



### User Impact
The entity's calculated risk score will slightly increase because we
update the normalisation divisor from 261.2 to 2.5924.




### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
2024-06-06 14:30:38 +02:00
Miriam
22155aefdc
[ObsUX] [Infra] Add missing metrics to Docker container view (#184245)
Closes https://github.com/elastic/kibana/issues/183354

Metric section

![image](0d85da8b-a651-4557-ac09-e3eee2573c42)

How to test
- The feature is under a FF, on inventory page go to settings and enable
Container view
- In containers inventory, select a docker container, you find one,
filter by any `docker.` field. Click on a container.
- Container details page should be shown Network and DiskIO charts, as
well as CPU and Memory, on Metrics section
2024-06-06 12:59:29 +01:00
Ido Cohen
a03f3bce7d
[Cloud Security] FTRs for Accessing Pages with Custom Roles (#184622) 2024-06-06 13:06:39 +03:00
Lisa Cawley
90d1bc6388
[DOCS] Stack Management Alerts app (#184384) 2024-06-05 16:12:54 -07:00
Rickyanto Ang
3b88219aed
[Cloud Security] Update CSP Version to 1.8.1 for Test (#184492)
## Summary

With 8.13 released, we want to make sure our CSP is using the latest CSP
version for our test environment
2024-06-05 14:12:30 -07:00
Jen Huang
dceae3e5c3
[UII] Remove references to .fleet-servers index (#183868)
## Summary

Resolves https://github.com/elastic/kibana/issues/173537.

This PR removes all usages and references to `.fleet-servers` index.

A small refactoring was done that moves helper functions in enrollment
settings API to fleet server services. It's a better place for them
because they are related to fleet server anyway. With this change, we
now use these functions to calculate the readiness of Fleet
(`hasFleetServers` was previously reading on `.fleet-servers` index).

The readiness of a fleet server is defined as at least one online agent
enrolled into an agent policy that contains a fleet server policy.

### Changes in Security Solution (@paul-tavares)

- Updated `enableFleetServerIfNecessary()` utility _(used for testing,
dev)_ to not use `.fleet-servers` index and instead write a Agent record
to the `.fleet-agents` index for Fleet Server. This record writing is
skipped when running tests in serverless mode, instead,
`xpack.fleet.internal.fleetServerStandalone=true` is added to mimic
skipping checks for Fleet Server in real serverless.


### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

---------

Co-authored-by: Paul Tavares <paul.tavares@elastic.co>
2024-06-05 13:53:16 -07:00
Tiago Costa
bff17fe57f
skip flaky suite (#184853) 2024-06-05 20:38:46 +01:00
Ying Mao
74d1430305
Revert "[Response Ops][Task Manager] Emitting metrics when metrics are reset (#184846)
## Summary

Reverted
557633456c
from deploy@1717401777 as part of emergency release. This PR is
following the emergency release guidelines to:

`In a separate PR, the fix should be "frontported" to main by manually
cherry-picking the commit.`
2024-06-05 13:40:00 -04:00
Nicolas Chaulet
1836f6e636
[Fleet] Write space aware .fleet-policies and .fleet-enrollment-token… (#184676) 2024-06-05 12:52:11 -04:00
Tiago Costa
db3f866a0d
skip flaky suite (#184619) 2024-06-05 16:06:18 +01:00
Ignacio Rivas
af1d12cc8a
[Index management] Add some time to guarantee execution before deletion (#183980) 2024-06-05 10:24:44 +02:00
jennypavlova
3793d2f04a
[Infra] Add asset details flyout test for the initial container view (#184746)
Closes #183899 

## Summary

This PR adds asset details flyout test for the initial container view.
2024-06-05 10:11:52 +02:00
Mashhur
d750aca73f
Telemetry collection support for Logstash agent driven monitoring. (#184480)
## Summary
Telemetry collection support for Logstash agent driven monitoring.
This change also guarantees to collect metricbeat
monitoring metrics when no cluster UUID found in the event.

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-06-04 16:49:56 -07:00
Achyut Jhunjhunwala
59bc79c170
[Dataset Quality]Add logic to display spark plot on degraded fields (#184514)
## Summary

Closes: https://github.com/elastic/kibana/issues/183604

The PR adds a Spark Plot to the Degraded Fields Table in the Dataset
Quality Flyout

## Screenshot

<img width="818" alt="image"
src="45636e33-e6e8-4096-af2f-8cc42b1bd2e6">
2024-06-04 09:43:43 -07:00
Vitalii Dmyterko
c89ee65c70
[Security Solution][Detection engine] fixes flaky query rule execution test (#184597)
## Summary

- `should query and get back expected alert structure when it is a alert
on a alert` test was flaky as it asserted index name with a current
date. If it runs near midnight, when date changes it could lead to
failed index name assertions. Converted that tests to use regexp, so
date won't be asserted.
- migrated query test file from kibana expect to jest expect
- addresses flaky tests issue
  - https://github.com/elastic/kibana/issues/174805
  - https://github.com/elastic/kibana/issues/177101
  - https://github.com/elastic/kibana/issues/173342
2024-06-04 16:06:17 +01:00
Philippe Oberti
506ff4cbb6
[Security Solution][Expandable flyout] - replace advanced settings with feature flag (#184169) 2024-06-04 10:04:19 -05:00
Angela Chuang
7ab3676af4
[SecuritySolution] Relocate endpoint tests (#183611)
## Summary

https://github.com/elastic/kibana/issues/181683

This PR moves 

1. x-pack/test/security_solution_endpoint_api_int to
`x-pack/test/security_solution_api_integration/test_suites/security_solution_endpoint_api_int`
2. x-pack/test/security_solution_endpoint to
`x-pack/test/security_solution_api_integration/test_suites/security_solution_endpoint`
3. x-pack/test/timeline to
`x-pack/test/security_solution_api_integration/test_suites/investigation/timeline`

### To test:

1. ```cd x-pack/test/security_solution_api_integration```

2. ```node ../../../scripts/functional_tests_server.js --config
./test_suites/security_solution_endpoint/serverless.endpoint.config.ts```

Once the server is launched (you might need Docker to run the serverless
tests), open another terminal, go to the same path, and execute the
command appears in the original:

The command should look like: ```node
../../../scripts/functional_test_runner
--config=test_suites/security_solution_endpoint/serverless.endpoint.config.ts```

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-06-04 13:04:01 +01:00
Marco Antonio Ghiani
01a29261a1
[Observability] Set observable initial value to prevent rerender (#184652)
## 📓 Summary

Adding an initial value to the `useObservable` call prevents its value
from flapping and rendering twice the whole child application.

This issue is similar to what was reported in #181968 and also its
solution lies on a similar error.

I added a functional test for both stateful/serverless environments to
assert this behaviour doesn't break anymore.

---------

Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2024-06-04 09:58:23 +02:00
Umberto Pepato
a8b2eef204
[ResponseOps][Alerts] Update stack alerts page feature labels and filters (#184305)
## Summary

Updates the alerts table feature cell renderers and feature filters
according to #183615.

## To verify

1. Create some Stack, Stack Monitoring, and ML rules that fire alerts
2. Navigate to `Stack Management` > `Alerts`
3. Verify that the feature quick filter (found in the <img width="22"
alt="image"
src="847c3fc4-1c23-4846-ae32-d66b10de7ad5">
menu of the KQL bar) respects the changes outlined in points 1 and 2 of
#183615
4. Verify that the alerts from Stack rules show a `Stack` label in the
table
6. Verify that the alerts from Stack Monitoring rules show a `Stack
Monitoring` label in the table
7. Verify that filtering by `Stack rule types` shows only Stack and ML
alerts in the table
8. Verify that filtering by `Observability rule types` doesn't show
Stack Monitoring alerts
9. Verify that filtering by `Stack Monitoring rule types` only shows
Stack Monitoring alerts

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2024-06-04 09:18:02 +02:00
Quynh Nguyen (Quinn)
d720a9bff3
[ML] Adds ES|QL support for field statistics table in Discover (#180849)
## Summary

Follow up of https://github.com/elastic/kibana/pull/179098. Adds ES|QL
for support field statistics table in Discover

<img width="1728" alt="Screenshot 2024-05-09 at 13 59 29"
src="ede8bcc5-2fc6-4a0a-8e3e-993294bbeafd">


09a1feeb-cff6-4b8b-bf82-bb930413f215



2acd78ec-1856-406e-a8f9-f6b2a26f8d81



### Checklist

Delete any items that are not applicable to this PR.

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)


### Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.

When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:

| Risk | Probability | Severity | Mitigation/Notes |

|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces&mdash;unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes&mdash;Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |


### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

---------

Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Davis McPhee <davis.mcphee@elastic.co>
2024-06-03 14:34:01 -05:00
Alexi Doak
1c56c518b7
[ResponseOps] Flapping recovered alerts do not show up as active in the alerts table (#184255)
Resolves https://github.com/elastic/kibana/issues/183735

## Summary

This bug is caused by event log docs not being written for "pending
recovered" alerts on rules with "notify on every run" and "throttle" or
notify when set to null. In this PR I remove the logic causing this in
`xpack/plugins/alerting/server/lib/get_alerts_for_notification.ts` so
now these alerts will be logged in the event log and show up in the ui.

### Checklist

- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios


### To verify

- Create an ES Query rule without actions, or with `notifyWhen` set to
"notify on every run" or "throttle". Get it to start flapping.
- Let the flapping alert start to recover and verify it's still reported
as active in the UI until it's been recovered for X executions (the
default is 4 if you don't mess with your flapping settings).

Ex. of what to expect
<img width="1461" alt="Screen Shot 2024-05-29 at 10 53 57 AM"
src="144dc22b-a1c7-460f-b430-a6d49771c9bd">
2024-06-03 11:28:11 -07:00
Gloria Hornero
9d84341f6b
[Security Solution] Changes default role for serverless from admin to platform_engineer (#183608)
## Summary

Addresses: https://github.com/elastic/kibana/issues/184140

In this PR we are addressing the short-term solution of
https://github.com/elastic/kibana/issues/184135 where we are moving from
using for serverless `admin` role for MKI environments and
`system_indices_superuser` for stateless environments to
`platform_engineer`.

To make the above change happen, some adjustments have been introduced
to make the tests work:

### Rename of custom indexes

The `admin` or `system_indices_superuser` has access to any index on the
system, but built-in roles in security serverless projects just have
access to some or all the security default indexes, so they cannot
access custom indexes.

To make the tests work with the `platform_engineer` all the custom
indexes that we use for testing purposes have been renamed to match
ours.

### `waitForTheRuleToBeExecuted` removed from some tests

`waitForTheRuleToBeExecuted` waits for the rule status to be `succeeded`
but the `platform_engineer` does not have access to all the indexes, so
instead the final status result will be `warning` so this method will
timeout.

Originally that method was created in order to make sure that some
alerts are going to be displayed but, after it, we are calling
`waitForAlertsToPopulate` that makes sure that we have alerts.

### Some tests have been skipped from serverless executions
- Entity analytics tests exercising the legacy risk score have been
removed from serverless executions, since it is not used on serverless
projects.
- Maintenance window callout test has been skipped in serverless due to
the following reason: https://github.com/elastic/kibana/issues/184160
- One of the users tabs test has been skipped as well due to the
following reason: https://github.com/elastic/kibana/issues/184201

---------

Co-authored-by: Devin Hurley <devin.hurley@elastic.co>
Co-authored-by: Georgii Gorbachev <banderror@gmail.com>
2024-06-03 15:15:46 +02:00
Sébastien Loix
eee34b2486
[Stateful sidenav] Cleanup iteration 2 (#184367) 2024-06-03 12:59:27 +01:00
elena-shostak
b4238544ae
[Spaces] Telemetry for space solution property (#184192)
## Summary

Added telemetry for space solution property.

### How to test
```
# Create a couple of spaces with solution
POST kbn:/api/spaces/space 
{
  "name": "space with solution",
  "id": "my-space-solution-1",
  "description": "a description",
  "color": "#5c5959",
  "solution": "search",
  "disabledFeatures": []
}

POST kbn:/api/spaces/space 
{
  "name": "space with solution",
  "id": "my-space-solution-2",
  "description": "a description",
  "color": "#5c5959",
  "solution": "search",
  "disabledFeatures": []
}

POST kbn:/api/spaces/space 
{
  "name": "space with solution",
  "id": "my-space-solution-3",
  "description": "a description",
  "color": "#5c5959",
  "solution": "security",
  "disabledFeatures": []
}

# Get stats data
POST kbn:/internal/telemetry/clusters/_stats
{
  "unencrypted": true, "refreshCache": true
}
```


### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

### For maintainers

- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

__Fixes: https://github.com/elastic/kibana/issues/183641__

## Release Note
Added telemetry for space solution property.

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2024-06-03 13:47:37 +02:00
Pierre Gayvallet
dea26c6450
Add http2 support for Kibana server (#183465)
## Summary

Part of https://github.com/elastic/kibana/issues/7104

Add support for `http2` to the Kibana server. `http2` can be enabled by
setting `server.protocol: http2` in the Kibana config file.

*Note: by default, enabling `http2` requires a valid `h2c`
configuration, meaning that it can only run over HTTPS with TLS1.2+*

```yaml
## kibana.yaml
server.protocol: http2
server.ssl.enabled: true
server.ssl.key: path/to/key
server.ssl.certificate: path/my/cerf
```

## What is this PR doing

### Add HTTP2 support for the Kibana server

#### - Plug http2 to the Kibana server 

Even if HAPI was never officially updated to really support HTTP2,
node's `http`/`https`/`http2` modules are compatible enough to be able
to just instantiate an http2 server/listener and provide it to HAPI "as
a plain https listener". There were some tweaks to do (mostly silencing
a few warnings that HAPI was causing by sending http2-illegal headers
such as `Connection`), but overall, it went smoothly.

#### - Add config validation

By default, Kibana will require a valid `h2c` configuration to accept
enabling `http2`. It means that TLS must be enabled and that TLS1.2+
should at least be in the list of supported SSL protocols
(`server.ssl.supportedProtocols`). Note that default value of this
setting includes TLS1.2 and 1.3.

#### - Add escape hatch to run `h2` without `h2c`

In some situations, it may be required to enable http2 without a valid
`h2c` configuration. Kibana supports it, by setting
`server.http2.allowUnsecure` to `true`.

(*Note, however, that if http2 is enabled without TLS, ALPN protocol
negotiation won't work, meaning that most http2 agents/clients will fail
connecting unless they're explictly configured to use http2.*)

### Add documentation about this new feature

#### - Update the user-facing doc about this new `server.protocol`
setting

Update the user-facing Kibana settings documentation to include this
`http.protocol` setting (and refer to `server.http2.allowUnsecure`)

**Note: this setting, and this feature, are considered as experimental**

### Adapt our dev tooling to support running Kibana with http2 enabled

#### - Add a `--http2` flag to the dev CLI

Enabling this flag will add the proper configuration settings to run
Kibana with `http2` enabled in an (almost) valid `h2c` configutation.

*Note: when using this flag, even if listening on the same port, the
Kibana server will be accessible over https, meaning that you need to
use https in your browser to access it. Aka `http://localhost:5601`
won't work, you need to use `https://localhost:5601`. Also, we're using
the self-signed dev certificates, meaning that you must go though the
scary warning of your browser*

#### - Implement an http2-compatible base-path proxy

The current base path proxy is based on `hapi` and `hapi/h2o2`. I tried
for a bunch hours trying to hack around to make it work with http2
proxying, but ultimately gave up and implemented a new version from
scratch.

Note that with some additional efforts, this new http2 basepath proxy
could probably fully replace the existing one and be used for both http1
and http2 traffic, but it's an optimization / refactoring that did not
feel required for this PR.

### Adapt the FTR to run suites against http2

#### - Add support to run FTR test suite against an h2c-enabled Kibana

Note that with ALPN, clients using http1 should be (and are) able to
communicate with http2 Kibana, given h2c/alpn allows protocol
negitiation. So adapting our FTR tooling was not really about making it
work with http2 (which worked out of the box), but making it work with
**the self signed certifcates we use for https on dev mode**

Note that I'm not a big fan of what I had to do, however, realistically
this was the only possible approach if we want to run arbitrary test
suites with TLS/HTTP2 enabled without massively changing our FTR setup.

Operations and QA, feel free to chime in there, as this is your
territory.

#### - Change some FTR test suites to run against an HTTP2-enabled
server

I added a quick `configureHTTP2` helper function to take any "final" FTR
suite config and mutate it to enable `http2`. I then enabled it on a few
suites locally, to make sure the suites were passing correctly.

I kept two suites running with http2 enabled:
- the `console` oss functional tests
- the `home` oss functional tests

We could possibly enable it for more, but we need to figure out what
kind of strategy we want on that matter (see below)

## What is this pull request NOT doing

#### - Making sure everything works when HTTP2 is enabled

I navigated the applications quite a bit, and did not see anything
broken, however I obviously wasn't able to do a full coverage. Also, the
self-signed certificate was a huge pain to detect issues really caused
by http2 compared to issues because the local setup isn't valid `h2c`.

In theory though (famous last words) anything not doing http/1.1
specific hacks such as bfetch should work fine with http2, given that
even if using non-http2 clients, ALPN should just allow to fallback to
http/1.x (this part was tested)

#### - Enabling HTTP2 by default

PR isn't doing it for obvious reasons. 

#### - Enabling HTTP2 for all FTR suites

First of all, it's not that easy, because it requires adapting various
parts of the config (and even some var env...), and we don't have any
proper way to override config "at the end". For instance, if you add the
http2 config on a top level config (e.g. the oss functional one that is
reuse by the whole world - learned the hard way), it won't work because
higher-level configs redefined (and override) the `browser` part of the
config, loosing the settings added to run the browser in insecure mode.

Secondly, I'm not sure we really need to run that many suites with http2
enabled. I learned working on that PR that we only have like one suite
where https is enabled for the Kibana server, and I feel like it could
be fine to have the same for http2. In theory it's just a protocol
change, unless parts of our apps (e.g. bfetch) are doing things that are
specific to http/1.1, switching to http2 should be an implementation
detail.

But I'd love to get @elastic/kibana-operations and @elastic/appex-qa
opinion on that one, given they have more expertise than I do on that
area.

- Running performances tests

We should absolutely run perf testing between http/1.1 over https and
http/2, to make sure that it goes into the right directly (at least in
term of user perceived speed), but I did not do it in the scope of this
PR (and @dmlemeshko is on PTO so... 😅)

## Release Note

Add support for `http2` to the Kibana server. `http2` can be enabled by
setting `server.protocol: http2` in the Kibana config file.

Note: by default, enabling `http2` requires a valid `h2c` configuration,
meaning that it can only run over HTTPS with TLS1.2+

Please refer to the Kibana config documentation for more details.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-06-03 09:34:13 +02:00
Ahmad Bamieh
f1c854b9db
[i18n][system upgrade] Upgrade Intl Packages from v2 to v6 (#179506)
## Summary

Upgrading intl packages from v2 to v6

### Packages upgrade:

- [x] Add @formatJS packages
- [x] `react-intl` Upgraded
- [x] `intl-messageformat` Upgraded
- [x] `intl-format-cache` removed
- [x] `intl-relativeformat` removed
- [x] `intl-messageformat-parser` removed

### Todo list:
- [x] Refactor HTML tags
- [x] Refactor Upgrade tags
- [x] Refactor `kbn-i18n`
- [x] Refactor `kbn-i18n-react`
- [x] Refactor `FormattedRelative` to `FormattedRelativeTime`
- [x] Refactor polyfills
- [x] Refactor IntlShape types
- [x] Rewrite Providers
- [x] Rewrite tests using i18n
- [x] Removed current pseudolocale implementation (tracker:
https://github.com/elastic/kibana/issues/180244)
- [x] Fix jest tests using rendered `Provider`
- [x] Remove no longer valid i18n packages documentation (tracker:
https://github.com/elastic/kibana/issues/180259)

Closes https://github.com/elastic/kibana/issues/178968
Closes https://github.com/elastic/kibana/issues/38642


## Notes to code reviewers
For team other than the core team, please review your plugins code
changes by filtering files by codeowners.


### Test Snapshot updates
Most of the changes are refactors of renamed functions and changed ICU
syntax.

The main updates are snapshot changes where `FormattedMessage` is now
memoized so snapshots capturing the html tree needed to be updated to
use `<Memo(MemoizedFormattedMessage)` instead of `<FormattedMessage`


### ICU now supports HTML tags:
before:
```
<FormattedMessage
  defaultMessage="To buy a shoe, { link } and { cta }"
  values={{
    link: (
      <a class="external_link" target="_blank" href="https://www.shoe.com/">
        visit our website
      </a>
    ),
    cta: <strong class="important">eat a shoe</strong>,
  }}
/>
```
after:
```
<FormattedMessage
  defaultMessage="To buy a shoe, <a>visit our website</a> and <cta>eat a shoe</cta>"
  values={{
    a: msg => (
      <a class="external_link" target="_blank" href="https://www.shoe.com/">
        {msg}
      </a>
    ),
    cta: msg => <strong class="important">{msg}</strong>,
  }}
/>
```

### Escape character to prevent ICU parsing changed from double slashes
to single quotes:
before: `\\{escaped\\}`
after: `'{escaped}'`

### No need for Intl Shape
the new packages under formatJS are written in typescript and come with
types support out of the box so no need to set types when using i18n.

Renamed `InjectedIntlProps` with `WrappedComponentProps`.
Removed `prop-types` and `intlShape` in favor of `IntlShape`.

### FormattedRelative has been renamed to FormattedRelativeTime and its
API has changed significantly. See
[FormattedRelativeTime](https://formatjs.io/docs/react-intl/upgrade-guide-3x#formattedrelativetime)
for more details.

### All tags specified must have corresponding values and will throw
error if it's missing
All tags are now parsed and expected to be formatted properly (all
opened tags must be closed).

To skip this check you can use the `ignoreTag: true` property

```
i18n.translate('xpack.apm.agentConfig.captureJmxMetrics.description', {
      defaultMessage: 'This is not an HTML tag <JMX object name pattern>' +
      ignoreTag: true,
    }),
```

**When do I use ignore tags?**

If your message has HTML tags, it is preferred not to ignore the Tag to
have some string verification that the html tags you are adding are
properly formatted and closed.

If it the text between brackets is not an HTML tag and it is just a
fomat preference then using `ignoreTag` makes sense.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
2024-06-02 16:50:33 +03:00
Lukas Olson
f20b8d5d14
[data.search] Improve SearchSource types (part 1) (#184531)
## Summary

Minor improvements to types in `SearchSource`.

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
2024-06-01 08:23:55 -07:00
Ying Mao
557633456c
[Response Ops][Task Manager] Emitting metrics when metrics are reset (#184592)
## Summary

This updates the task manager metrics aggregator to collect and emit
metrics when a `reset$` event is observed.

The `/api/task_manager/metrics` route subscribes to and saves the latest
task manager metrics and immediately returns the latest metrics when the
API is accessed. At a minimum, metrics are collected and emitted at
every polling interval (every 3 seconds). Usually emission is more
frequent than this because we emit metrics events every time a task run
completes.

Under normal circumstances, when the agent is configured to collect from
the API once every 10 seconds, this is what happens

```
00:00:00 metrics$.subscribe(({errors: 3}) => lastMetrics = metrics) - metrics emitted and saved
00:00:03 metrics$.subscribe(({errors: 4}) => lastMetrics = metrics) - metrics emitted and saved
00:00:05 API called with reset=true, return lastMetrics, metrics reset to 0
00:00:06 metrics$.subscribe(({errors: 1}) => lastMetrics = metrics) - metrics emitted and saved
00:00:09 metrics$.subscribe(({errors: 2}) => lastMetrics = metrics) - metrics emitted and saved
00:00:10 API called with reset=true, return lastMetrics, metrics reset to 0
```

We can see that the metrics are reset and then by the time the next
collection interval comes around, fresh metrics have been emitted.

We currently have an issue where the API is collected against twice in
quick succession. Most of the time, this leads to duplicate metrics
being collected.

```
00:00:00:00 metrics$.subscribe(({errors: 3}) => lastMetrics = metrics) - metrics emitted and saved
00:00:03:00 metrics$.subscribe(({errors: 4}) => lastMetrics = metrics) - metrics emitted and saved
00:00:05:00 API called with reset=true, return lastMetrics, metrics reset to 0 
00:00:05:01 API called with reset=true, return lastMetrics, metrics reset to 0 - this is a duplicate
00:00:06:00 metrics$.subscribe(({errors: 1}) => lastMetrics = metrics) - metrics emitted and saved
00:00:09:00 metrics$.subscribe(({errors: 2}) => lastMetrics = metrics) - metrics emitted and saved
```

However sometimes, this leads to a race condition that leads to
different metrics being collected.

```
00:00:00:00 metrics$.subscribe(({errors: 3}) => lastMetrics = metrics) - metrics emitted and saved
00:00:03:00 metrics$.subscribe(({errors: 4}) => lastMetrics = metrics) - metrics emitted and saved
00:00:05:00 API called with reset=true, return lastMetrics, metrics reset to 0
00:00:05:01 metrics$.subscribe(({errors: 1}) => lastMetrics = metrics) - metrics emitted and saved
00:00:05:02 API called with reset=true, return lastMetrics, metrics reset to 0
00:00:06:00 metrics$.subscribe(({errors: 1}) => lastMetrics = metrics) - metrics emitted and saved
00:00:09:00 metrics$.subscribe(({errors: 2}) => lastMetrics = metrics) - metrics emitted and saved
```

With this PR, on every reset, we'll re-emit the metrics so so even in
the face of the duplicate collection, we won't be emitting duplicate
metrics. After this is deployed, we should not need to exclude
`kubernetes.container.name :"elastic-internal-init-config"` from the
dashboards

```
00:00:00:00 metrics$.subscribe(({errors: 3}) => lastMetrics = metrics) - metrics emitted and saved
00:00:03:00 metrics$.subscribe(({errors: 4}) => lastMetrics = metrics) - metrics emitted and saved
00:00:05:00 API called with reset=true, return lastMetrics, metrics reset to 0 
00:00:05:00 metrics$.subscribe(({errors: 0}) => lastMetrics = metrics) - metrics emitted and saved
00:00:05:01 API called with reset=true, return lastMetrics, metrics reset to 0
00:00:05:01 metrics$.subscribe(({errors: 0}) => lastMetrics = metrics) - metrics emitted and saved
00:00:06:00 metrics$.subscribe(({errors: 1}) => lastMetrics = metrics) - metrics emitted and saved
00:00:09:00 metrics$.subscribe(({errors: 2}) => lastMetrics = metrics) - metrics emitted and saved
```
2024-05-31 14:05:09 -04:00
Yara Tercero
a4577070ac
[Detection Engine][Cypress] - Reduce exceptions flake - alerts_table_flow (#184383)
## Summary

Reduce flake for tests that check exceptions auto populate with alert
data. I moved to primarily checking what is being sent to the API after
the expected user interaction as a more solid way of confirming that the
flow is working as expected. Before we were checking that every value we
expect is being shown in the UI, which leads to flake depending on if
there are unexpected re-renders. There certainly are improvements to
make code side vs. test side, but this is a way to get our tests back up
and running.
2024-05-31 10:52:31 -07:00
Yara Tercero
132bc97a85
[Detection Engine] Update internal endpoints to indicate them as such (#184326)
While the Rules Management team was researching current status of OpenAPI migration in Security Solution, they found two of our endpoints that are /internal/... but marked as access:public. Updating the routes to be as intended - access: internal.
2024-05-31 10:25:45 -07:00
Tiago Costa
a78522d5c1
Revert "skip flaky suite (#179353)"
This reverts commit 0f8befbe06.
2024-05-31 17:44:21 +01:00
Tiago Costa
0f8befbe06
skip flaky suite (#179353) 2024-05-31 17:43:13 +01:00
Catherine Liu
aa109676fa
[Tests] Add test coverage for handling multiple data views (#184460)
## Summary

Tests extracted from #180425.

This adds some smoke tests coverage for supporting multiple data views
and how dashboards/visualizations interact with the
`courier:ignoreFieldIfNotInIndex` advanced setting in Dashboard,
Controls, Lens, and Maps.

Flaky test runner x 50:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6160

### Checklist

Delete any items that are not applicable to this PR.

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)


### Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.

When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:

| Risk | Probability | Severity | Mitigation/Notes |

|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces&mdash;unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes&mdash;Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |


### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
2024-05-31 08:36:40 -07:00
Kibana Machine
45a73a7f29 skip failing test suite (#184585) 2024-05-31 11:10:07 -04:00
Julia Bardi
464f797a73
[Fleet] Prevent concurrent runs of Fleet setup (#183636)
Closes https://github.com/elastic/ingest-dev/issues/3346

- [x] Unit and integration tests are created or updated
- [x] Turn down info logging

The linked issue seems to be caused by multiple kibana instances running
Fleet setup at the same time, trying to create the preconfigured cloud
policy concurrently, and in case of failures, the agent policy is left
with a revision with no inputs, this way preventing fleet-server to
start properly.

See the concurrent errors in the logs:
https://platform-logging.kb.us-west2.gcp.elastic-cloud.com/app/r/s/tUpMP

This fix introduces a `fleet-setup-lock` SO type, which is used to
create a document as a lock by Fleet setup, and is deleted when the
setup is completed. Concurrent calls to Fleet setup will return early if
this doc exists.

To verify:
Run the test `./run_fleet_setup_parallel.sh` from local kibana, and
verify the generated logs that only one of them ran Fleet setup.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-05-31 16:38:51 +02:00
Julia Bardi
3701edc6a2
[Fleet] Fix bulk install package with force flag when package is already installed (#184580)
## Summary

Closes https://github.com/elastic/kibana/issues/184491

Small fix for bulk install package API force flag, to allow reinstalling
a package if already installed.

To verify:
```
# Install Apache integration 
POST kbn:/api/fleet/epm/packages/apache

# Install older version with force
POST kbn:/api/fleet/epm/packages/_bulk
{
  "packages": [
    {
      "name": "apache",
      "version": "1.17.2"
    }
  ],
  "force": true
}

# Expect 1.17.2 to be installed
{
  "items": [
    {
      "name": "apache",
      "version": "1.17.2",
      "result": {
        "assets": [
          ...
        ],
        "status": "installed",
        "installType": "install",
        "installSource": "registry"
      }
    }
  ],
```

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
2024-05-31 16:38:02 +02:00
Alexi Doak
839d611557
[ResponseOps] Flaky test x-pack/test/alerting_api_integration/spaces_only/tests/action_task_params/migrations·ts (#184473)
Resolves https://github.com/elastic/kibana/issues/154358

## Summary

I couldn't get this to fail, but it's using `esArchiver` which is known
to be flaky. I used the info in this issue, [Proactively improve
robustness of FTR tests that use esArchiver
#161882](https://github.com/elastic/kibana/issues/161882), to maybe help
reduce the flakiness.

Flaky test runner:
[run
1](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6163)
x 50
[run
2](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6164)
x 200
[run
3](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6165)
x 200
2024-05-31 07:20:12 -07:00
Vitalii Dmyterko
83861dfaa9
[Security Solution][Detection Engine] fix broken serverless ES|QL tests (#184581)
## Summary

- https://github.com/elastic/kibana/pull/183096 PR disabled bfetch
requests for Serverless, leading to failure of multiple serverless ES|QL
tests that rely on intercepting `bsearch` request
- addresses:
  -  https://github.com/elastic/kibana/issues/184558
  -  https://github.com/elastic/kibana/issues/184556
  - https://github.com/elastic/kibana/issues/184557
2024-05-31 06:28:17 -07:00
Julia Bardi
3ec981d036
[Fleet] include inactive agents in agent policy agent count (#184517)
## Summary

Closes https://github.com/elastic/kibana/issues/180356

Include inactive agents in agent policy agent count to make it
consistent with the check that prevents deleting an agent policy.

To verify:
- create an agent policy with a short inactivity timeout e.g. 10 seconds
- enroll an agent and wait to become inactive
- go to agent policy details and verify the agent count shows 1 agent
- verify that the delete agent policy modal has a consistent count

<img width="1376" alt="image"
src="c4ca8084-b9ae-49a6-956e-cdc58217da03">
<img width="857" alt="image"
src="0d007494-d26f-48da-8c99-388ce74946d6">

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
2024-05-31 13:54:41 +02:00
Kibana Machine
690cc68a86 skip failing test suite (#184558) 2024-05-31 07:25:17 -04:00