## Summary
Updating the ES client to 9.0.
Resolves#116102
## What changes?
**Breaking change**: `body` has been removed.
Most of the changes are about bringing all the content inside the body
as a root attribute to the API params:
```diff
const response = await client.search({
index: 'test',
- body: {
query: {
match_all: {}
}
- }
})
```
For this reason, enabling the "Hide whitespace changes" option when
reviewing is recommended.
Some exceptions to this rule:
* Bulk APIs replace the `body` array with `operations` array (direct
replacement)
* Index Put Settings API replace `body` array with `settings` (direct
replacement)
* Msearch replaces the `body` array with `searches` array (direct
replacement)
* Document Index API replaces `body` with `document` (direct
replacement)
* Create Repository replaces `body` with `repository` (direct
replacement)
Because of a known issue in the client
(https://github.com/elastic/elasticsearch-js/issues/2584), there's still
an escape hatch to send data in the body in case the specific use case
requires it via `// @ts-expect-error elasticsearch@9.0.0
https://github.com/elastic/elasticsearch-js/issues/2584`, but it
shouldn't be abused because we lose types. In this PR we've used it in
those scenarios where we reuse the response of a GET as the body of a
PUT/POST.
### Other changes
* `estypes` can be imported from the root of the library as `import type
{ estypes } from '@elastic/elasticsearch';`
* `estypesWithBody` have been removed
* `requestTimeout`'s 30s default has been removed in the client. This PR
explicitly adds the setting in all client usages.
### Identify risks
- [x] The client places unknown properties as querystring, risking body
params leaking there, and causing 400 errors from ES => Solved by
forcing `body` usage there via `// @ts-expect-error elasticsearch@9.0.0
https://github.com/elastic/elasticsearch-js/issues/2584`. The next
version of the client will address this.
- [x] We need to run the MKI tests to make sure that we're not breaking
anything there =>
https://elastic.slack.com/archives/C04HT4P1YS3/p1739528112482629?thread_ts=1739480136.231439&cid=C04HT4P1YS3
---------
Co-authored-by: Gloria Hornero <gloria.hornero@elastic.co>
## Summary
Part of https://github.com/elastic/kibana/issues/195418
Gives `FROM` and `METADATA` autocomplete logic its own home 🏡
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### Identify risks
- [ ] As with any refactor, there's a possibility this will introduce a
regression in the behavior of FROM. However, all automated tests are
passing and I have tested the behavior manually and can detect no
regression.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
This PR adds a test config category to the scout reporting. This allows
us to distinguish between UI and API FTR tests.
A new property `testConfigCategory` has been added to all FTR configs
that don't already inherit it from a higher level config.
## Summary
This PR reworks saved query privileges to rely solely on a single global
`savedQueryManagement` privilege, and eliminates app-specific overrides.
This change simplifies the security model for users, fixes bugginess in
the saved query management UI, and reduces code complexity associated
with maintaining two separate security mechanisms (app-specific
overrides and global saved query management privileges).
### Background
Saved queries allow users to store a combination of KQL or Lucene
queries, filters, and time filters to use across various applications in
Kibana. Access to saved query saved objects are currently granted by the
following feature privileges:
```json
[
"feature_discover.all",
"feature_dashboard.all",
"feature_savedQueryManagement.all",
"feature_maps.all",
"feature_savedObjectsManagement.all",
"feature_visualize.all"
]
```
There is also a saved query management UI within the Unified Search bar
shared by applications across Kibana:
<img
src="https://github.com/user-attachments/assets/e4a7539b-3dd4-4d47-9ff8-205281ef50e3"
width="500" />
The way access to this UI is managed in Kibana is currently confusing
and buggy:
- If a user has `feature_discover.all` and `feature_dashboard.all` they
will be able to load and save queries in Discover and Dashboard.
- If a user has `feature_discover.all` and `feature_dashboard.read` they
will be able to load queries in both Discover and Dashboard, but only
save queries in Discover (even though they have write access to the SO,
and API access). Instead they have to navigate to Discover to save a
query before navigating back to Dashboard to load it, making for a
confusing and frustrating UX.
- Access to the UI is even more confusing in apps not listed in the
above feature privileges (e.g. alerting, SLOs). Some of them chose to
check one of the above feature privileges, meaning users who otherwise
should have saved query access won't see the management UI if they don't
also have the exact feature privilege being checked. Other apps just
always show the management UI, leading to bugs and failures when users
without one of the above feature privileges attempt to save queries.
### Existing improvements
In v8.11.0, we introduced a new ["Saved Query
Management"](https://github.com/elastic/kibana/pull/166937) privilege,
allowing users to access saved queries across all of Kibana with a
single global privilege:
<img
src="https://github.com/user-attachments/assets/ccbe79a4-bd0b-4ed6-89c9-117cc1f99ee2"
width="600" />
When this privilege is added to a role, it solves the
`feature_discover.all` and `feature_dashboard.read` issue mentioned
above. However, it does not fix any of the mentioned issues for roles
without the new privilege. We have so far postponed further improvements
to avoid a breaking change.
### Approach
To fully resolve these issues and migrate to a single global privilege,
these changes have been made:
- Remove saved query SO access from all application feature privileges
and instead only allow access through the global saved query management
privilege.
- Stop relying on application feature privileges for toggling the saved
query management UI, and instead rely on the global privilege.
To implement this with minimal breaking changes, we've used the Kibana
privilege migration framework. This allows us to seamlessly migrate
existing roles containing feature privileges that currently provide
access to saved queries, ensuring they are assigned the global saved
query management privilege on upgrade.
As a result, we had to deprecate the following feature privileges,
replacing them with V2 privileges without saved query SO access:
```json
[
"feature_discover.all",
"feature_dashboard.all",
"feature_maps.all",
"feature_visualize.all"
]
```
Each area of code that currently relies on any of these feature
privileges had to be updated to instead access `feature_X_V2` instead
(as well as future code).
This PR still introduces a minor breaking change, since users who have
`feature_discover.all` and `feature_dashboard.read` are now able to save
queries in Dashboard after upgrade, but we believe this is a better UX
(and likely the expected one) and worth a small breaking change.
### Testing
- All existing privileges should continue to work as they do now,
including deprecated V1 feature privileges and customized serverless
privileges. There should be no changes for existing user roles apart
from the minor breaking change outlined above.
- Check that code changes in your area don't introduce breaking changes
to existing behaviour. Many of the changes are just updating client UI
capabilities code from `feature.privilege` to `feature_v2.privilege`,
which is backward compatible.
- The `savedQueryManagement` feature should now globally control access
to saved query management in Unified Search for all new user roles.
Regardless of privileges for Discover, Dashboard, Maps, or Visualize,
new user roles should follow this behaviour:
- If `savedQueryManagement` is `none`, the user cannot see or access the
saved query management UI or APIs.
- If `savedQueryManagement` is `read`, the user can load queries from
the UI and access read APIs, but cannot save queries from the UI or make
changes to queries through APIs.
- If `savedQueryManagement` is `all`, the user can both load and save
queries from the UI and through APIs.
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [x] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
This PR risks introducing unintended breaking changes to user privileges
related to saved queries if the deprecated features have not been
properly migrated, and users could gain or lose access to saved query
management on upgrade. This would be bad if it happened, but not overly
severe since it wouldn't grant them access to any ES data they couldn't
previously access (only query saved objects). We have automated testing
in place to help ensure features have been migrated correctly, but the
scope of these changes are broad and touch many places in the codebase.
Additionally, the UI capabilities types are not very strict, and are
referenced with string paths in many places, which makes changing them
riskier than changing strictly typed code. A combination of regex
searches and temporarily modifying the `Capabilities` type to cause type
errors for deprecated privileges was used to identify references in
code. Reviewers should consider if there are any other ways that UI
capabilities can be referenced which were not addressed in this PR.
Our automated tests already help mitigate the risk, but it's important
that code owners thoroughly review the changes in their area and
consider if they could have unintended consequences. The Platform
Security team should also review this PR thoroughly, especially since
some changes were made to platform code around privilege handling. The
Data Discovery team will also manually test the behaviour when upgrading
existing user roles with deprecated feature privileges as part of 9.0
upgrade testing.
---------
Co-authored-by: Matthias Wilhelm <matthias.wilhelm@elastic.co>
Co-authored-by: Matthias Wilhelm <ankertal@gmail.com>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: “jeramysoucy” <jeramy.soucy@elastic.co>
## Summary
Updated the search plugin config endpoint to return just features and
kibana version, removing the call to the enterprise search node.
### Checklist
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fixes https://github.com/elastic/kibana/issues/197747.
Updates the `/internal/index-pattern-management/resolve_index/{query}`
route to properly handle `no_such_remote_cluster_exception` and return
`404` rather than `500` server error.
Adds unit tests for the route handler.
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Julia Rechkunova <julia.rechkunova@gmail.com>
## Summary
This PR aims at relocating some of the Kibana modules (plugins and
packages) into a new folder structure, according to the _Sustainable
Kibana Architecture_ initiative.
> [!IMPORTANT]
> * We kindly ask you to:
> * Manually fix the errors in the error section below (if there are
any).
> * Search for the `packages[\/\\]` and `plugins[\/\\]` patterns in the
source code (Babel and Eslint config files), and update them
appropriately.
> * Manually review
`.buildkite/scripts/pipelines/pull_request/pipeline.ts` to ensure that
any CI pipeline customizations continue to be correctly applied after
the changed path names
> * Review all of the updated files, specially the `.ts` and `.js` files
listed in the sections below, as some of them contain relative paths
that have been updated.
> * Think of potential impact of the move, including tooling and
configuration files that can be pointing to the relocated modules. E.g.:
> * customised eslint rules
> * docs pointing to source code
> [!NOTE]
> This PR has been auto-generated.
> Do not attempt to push any changes unless you know what you are doing.
> Please use
[#sustainable_kibana_architecture](https://elastic.slack.com/archives/C07TCKTA22E)
Slack channel for feedback.
#### 2 plugin(s) are going to be relocated:
| Id | Target folder |
| -- | ------------- |
| `@kbn/esql` | `src/platform/plugins/shared/esql` |
| `@kbn/esql-datagrid` | `src/platform/plugins/shared/esql_datagrid` |
#### 5 package(s) are going to be relocated:
| Id | Target folder |
| -- | ------------- |
| `@kbn/esql-ast` | `src/platform/packages/shared/kbn-esql-ast` |
| `@kbn/esql-editor` | `src/platform/packages/private/kbn-esql-editor` |
| `@kbn/esql-utils` | `src/platform/packages/shared/kbn-esql-utils` |
| `@kbn/esql-validation-autocomplete` |
`src/platform/packages/shared/kbn-esql-validation-autocomplete` |
| `@kbn/language-documentation` |
`src/platform/packages/private/kbn-language-documentation` |
<details>
<summary>Updated references</summary>
```
./.buildkite/scripts/steps/esql_generate_function_metadata.sh
./.buildkite/scripts/steps/esql_grammar_sync.sh
./.eslintignore
./.eslintrc.js
./.i18nrc.json
./docs/developer/plugin-list.asciidoc
./examples/esql_ast_inspector/public/plugin.ts
./examples/esql_validation_example/public/plugin.ts
./package.json
./packages/kbn-esql-validation-autocomplete/src/validation/validation.test.ts
./packages/kbn-monaco/BUILD.bazel
./packages/kbn-repo-packages/package-map.json
./packages/kbn-ts-projects/config-paths.json
./packages/kbn-ui-shared-deps-src/BUILD.bazel
./src/dev/storybook/aliases.ts
./src/platform/packages/private/kbn-esql-editor/jest.config.js
./src/platform/packages/private/kbn-language-documentation/jest.config.js
./src/platform/packages/private/kbn-language-documentation/package.json
./src/platform/packages/shared/kbn-esql-ast/jest.config.js
./src/platform/packages/shared/kbn-esql-utils/jest.config.js
./src/platform/packages/shared/kbn-esql-validation-autocomplete/README.md
./src/platform/packages/shared/kbn-esql-validation-autocomplete/jest.config.js
./src/platform/packages/shared/kbn-esql-validation-autocomplete/jest.integration.config.js
./src/platform/packages/shared/kbn-esql-validation-autocomplete/package.json
./src/platform/packages/shared/kbn-esql-validation-autocomplete/src/autocomplete/helper.ts
./src/platform/plugins/shared/esql/jest.config.js
./src/platform/plugins/shared/esql_datagrid/jest.config.js
./tsconfig.base.json
./yarn.lock
```
</details>
<details>
<summary>Updated relative paths</summary>
```
src/platform/packages/private/kbn-esql-editor/jest.config.js:12
src/platform/packages/private/kbn-esql-editor/tsconfig.json:2
src/platform/packages/private/kbn-language-documentation/jest.config.js:12
src/platform/packages/private/kbn-language-documentation/package.json:12
src/platform/packages/private/kbn-language-documentation/tsconfig.json:2
src/platform/packages/shared/kbn-esql-ast/jest.config.js:12
src/platform/packages/shared/kbn-esql-ast/tsconfig.json:2
src/platform/packages/shared/kbn-esql-utils/jest.config.js:12
src/platform/packages/shared/kbn-esql-utils/tsconfig.json:2
src/platform/packages/shared/kbn-esql-validation-autocomplete/jest.config.js:12
src/platform/packages/shared/kbn-esql-validation-autocomplete/jest.integration.config.js:12
src/platform/packages/shared/kbn-esql-validation-autocomplete/package.json:10
src/platform/packages/shared/kbn-esql-validation-autocomplete/package.json:11
src/platform/packages/shared/kbn-esql-validation-autocomplete/scripts/generate_function_definitions.ts:389
src/platform/packages/shared/kbn-esql-validation-autocomplete/tsconfig.json:2
src/platform/plugins/shared/esql/jest.config.js:12
src/platform/plugins/shared/esql/tsconfig.json:2
src/platform/plugins/shared/esql/tsconfig.json:7
src/platform/plugins/shared/esql_datagrid/jest.config.js:12
src/platform/plugins/shared/esql_datagrid/tsconfig.json:2
src/platform/plugins/shared/esql_datagrid/tsconfig.json:7
```
</details>
<details>
<summary>Script errors</summary>
```
```
</details>
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Rudolf Meijering <skaapgif@gmail.com>
## Summary
Make the following changes to the integrations page:
- Rename `enterprise_search` category to `search`
- Remove `Application Search` tag
- Remove the JSON tile entirely
fix https://github.com/elastic/kibana/issues/192052
## Summary
Internal APIs will be
[restricted](https://github.com/elastic/kibana/issues/163654) from
public access as of 9.0.0. In non-serverless environments, this breaking
change will result in a 400 error if an external request is made to an
internal Kibana API (route `access` option as `"internal"` or
`"public"`).
This PR allows API owners of non-xpack plugins to run their `ftr` API
integration tests against the restriction and adds examples of how to
handle it.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
Note to reviewers: The header needed to allow access to internal apis
shouldn't change your test output, with or without the restriction
enabled.
### How to test the changes work:
#### Non x-pack:
1. Set `server.restrictInternalApis: true` in `test/common/config.js`
2. Ensure your tests pass
#### x-pack:
1. Set `server.restrictInternalApis: true` in
`x-pack/test/api_integration/apis/security/config.ts`
2. Ensure the spaces tests pass
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
_The goal is to consolidate 2 existing test files with identical logic
into a single, deployment-agnostic test file:_
Files to be replaced:
- test/api_integration/apis/saved_objects_management/bulk_delete.ts
- test/api_integration/apis/saved_objects_management/bulk_get.ts
-
x-pack/test_serverless/api_integration/test_suites/common/saved_objects_management/bulk_delete.ts
-
x-pack/test_serverless/api_integration/test_suites/common/saved_objects_management/bulk_get.ts
New test file:
-
x-pack/test/api_integration/deployment_agnostic/apis/saved_objects_management/bulk_delete.ts
-
x-pack/test/api_integration/deployment_agnostic/apis/saved_objects_management/bulk_get.ts
The migration leverages the serverless test file as a basis for the new
deployment-agnostic test.
Necessary modifications are minimal and include FTR context provider &
its services to ensure compatibility across different environments.
Lastly new file has to be loaded into both the serverless and stateful
config files under
`x-pack/test/api_integration/deployment_agnostic/configs`
This approach ensures that the test logic remains consistent while
reducing redundancy and maintenance effort.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Follow-up to #190690
Most of API integration tests does not match the path pattern set in the
original PR (thanks @pheyos for catching it) and where not updated.
This PR updates `.eslintrc.js` with explicit patterns to lint
api_integration tests. Hopefully it is final change, but I rely on code
owners to double check it.
Most of the changes are trivial adjustments:
- duplicated before/after hooks `mocha/no-sibling-hooks`
- duplicated test titles `mocha/no-identical-title`
- async function in describe() `mocha/no-async-describe`
---------
Co-authored-by: Ash <1849116+ashokaditya@users.noreply.github.com>
## Summary
Moving common services to respective new homes.
Resolves: https://github.com/elastic/kibana/issues/188541
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
This PR enforces ESLint rules in FTR tests, in particular:
- `no-floating-promises` rule to catch unawaited Promises in
tests/services/page objects
_Why is it important?_
- Keep correct test execution order: cleanup code may run before the
async operation is completed, leading to unexpected behavior in
subsequent tests
- Accurate test results: If a test completes before an async operation
(e.g., API request) has finished, Mocha might report the test as passed
or failed based on incomplete context.
```
198:11 error Promises must be awaited, end with a call to .catch, end with a call to .then
with a rejection handler or be explicitly marked as ignored with the `void` operator
@typescript-eslint/no-floating-promises
```
<img width="716" alt="Screenshot 2024-08-20 at 14 04 43"
src="https://github.com/user-attachments/assets/9afffe4c-4b51-4790-964c-c44a76baed1e">
- recommended rules from
[eslint-mocha-plugin](https://www.npmjs.com/package/eslint-plugin-mocha)
including:
-
[no-async-describe](https://github.com/lo1tuma/eslint-plugin-mocha/blob/main/docs/rules/no-async-describe.md)
-
[no-identical-title.md](https://github.com/lo1tuma/eslint-plugin-mocha/blob/main/docs/rules/no-identical-title.md)
-
[no-sibling-hooks.md](https://github.com/lo1tuma/eslint-plugin-mocha/blob/main/docs/rules/no-sibling-hooks.md)
and others
Note for reviewers: some tests were skipped due to failures after
missing `await` was added. Most likely is a "false positive" case when
test is finished before async operation is actually completed. Please
work on fixing and re-enabling it
---------
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
### This PR introduces a new type of API integration tests in FTR:
deployment-agnostic
#### Test suite is considered deployment-agnostic when it fulfils the
following criteria:
**Functionality**: It tests Kibana APIs that are **logically identical
in both stateful and serverless environments** for the same SAML roles.
**Design**: The test design is **clean and does not require additional
logic** to execute in either stateful or serverless environments.
### How It Works
Most existing stateful tests use basic authentication for API testing.
In contrast, serverless tests use SAML authentication with
project-specific role mapping.
Since stateful deployments also support SAML, deployment-agnostic tests
**configure Elasticsearch and Kibana with SAML authentication in both
cases**. For roles, stateful deployments define 'viewer', 'editor', and
'admin' roles with serverless-alike privileges.
New `samlAuth` service has `AuthProvider` interface with 2 different
implementations: depending on environment context (serverless or
stateful) appropriate implementation is used. But it remains on service
level and hidden in test suite.
test example
```
export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
const samlAuth = getService('samlAuth');
const supertestWithoutAuth = getService('supertestWithoutAuth');
let roleAuthc: RoleCredentials;
let internalHeaders: InternalRequestHeader;
describe('GET /api/console/api_server', () => {
before(async () => {
roleAuthc = await samlAuth.createM2mApiKeyWithRoleScope('admin');
internalHeaders = samlAuth.getInternalRequestHeader();
});
after(async () => {
await samlAuth.invalidateM2mApiKeyWithRoleScope(roleAuthc);
});
it('returns autocomplete definitions', async () => {
const { body } = await supertestWithoutAuth
.get('/api/console/api_server')
.set(roleAuthc.apiKeyHeader)
.set(internalHeaders)
.set('kbn-xsrf', 'true')
.expect(200);
expect(body.es).to.be.ok();
const {
es: { name, globals, endpoints },
} = body;
expect(name).to.be.ok();
expect(Object.keys(globals).length).to.be.above(0);
expect(Object.keys(endpoints).length).to.be.above(0);
});
});
}
```
Please read
[readme](966822ac87/x-pack/test/api_integration/deployment_agnostic/README.md)
for more details and step-by-step guide. It should help migrating
existing serverless tests to deployment-agnostic, assuming requirements
are met.
### Examples
Deployment-agnostic tests:
```
x-pack/test/api_integration/deployment_agnostic/apis/console/spec_definitions.ts
x-pack/test/api_integration/deployment_agnostic/apis/core/compression.ts
x-pack/test/api_integration/deployment_agnostic/apis/painless_lab/painless_lab.ts
```
Configs to run it:
```
node scripts/functional_tests --config x-pack/test/api_integration/deployment_agnostic/oblt.serverless.config.ts
node scripts/functional_tests --config x-pack/test/api_integration/deployment_agnostic/search.serverless.config.ts
node scripts/functional_tests --config x-pack/test/api_integration/deployment_agnostic/security.serverless.config.ts
node scripts/functional_tests --config x-pack/test/api_integration/deployment_agnostic/stateful.config.ts
```
PR is a compact version of #188737 with reduced changes in existing
serverless tests.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: elena-shostak <165678770+elena-shostak@users.noreply.github.com>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>
Similar to how APM is shown as an integration as well, show the new
OTel-based flow on the integrations page so people find it from there as
well:
<img width="991" alt="Screenshot 2024-07-25 at 11 32 46"
src="https://github.com/user-attachments/assets/4d806ed1-4b01-4ac8-985c-0e59708fa4c6">
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Close https://github.com/elastic/kibana/issues/184836
Adds a new endpoint: `/api/_elu_history` for the purposes of tracking
load.
```ts
interface Response {
/**
* Event-loop utilization represented as an average of a number of collections as buckets
* @remark 3 load windows borrows from the `uptime` CLI tool on macOS and Linux, but is not necessarily 1m, 5m, 15m. The
* actual time range covered is determined by our collection interval (configured via `ops.interval`, default 5s)
* and the number of samples held in each window. So by default short: 15s, medium: 30s and long 60s.
*/
history: {
/** The average ELU for the short window */
short: number;
/** The average ELU for the medium window */
medium: number;
/** The average ELU for the long window */
long: number;
};
}
```
## How to test
Start Kibana locally (`yarn start --no-base-path`) and immediately run
the script below.
```bash
watch -n1 curl -s http://localhost:5601/api/_elu_history
```
Once Kibana starts responding you should see the `short > medium >
long`, then `short < medium < long` and eventually `short ~= medium ~=
long` if you let Kibana idle (basically, long needs to "lag" behind
short and medium).
## Questions
1. An alternative implementation exposes this directly via the
`/api/stats` endpoint as a new section under `event_loop_utilization`.
I'm assuming this approach is preferable.
2. Naming... What shall we call this endpoint?
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Part of https://github.com/elastic/kibana/issues/186530.
This PR sets the basis for allowing namespaced usage counters.
It relocates `usage-counters` SO type from `.kibana` to a dedicated
`.kibana_usage_counters`.
Furthermore, the original SO type is removed, and replaced by 2 separate
types:
* `server-counters`
* `ui-counters`
Note that these 2 steps are necessary if we want to leverage
`namespaces` property of the saved objects.
We can't currently update the `namespaceType: 'agnostic'` without
causing a migration.
Thus, these two types will be defined as `namespaceType: single`.
Up until now, UI counters were stored under a special `domainId:
uiCounter`.
This forced a workaround that consisted in storing `appName:eventName`
in the `counterName` property of the SO.
Having a dedicated SO type for them allows to store `appName` as
`domainId`, avoiding the need for a
[workaround](https://github.com/elastic/kibana/blob/main/src/plugins/usage_collection/common/ui_counters.ts).
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Reopening https://github.com/elastic/kibana/pull/186326 with my account,
non-internal PRs are just terrible to work with
---------
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@elastic.co>
## Summary
Part of https://github.com/elastic/kibana/issues/7104
Add support for `http2` to the Kibana server. `http2` can be enabled by
setting `server.protocol: http2` in the Kibana config file.
*Note: by default, enabling `http2` requires a valid `h2c`
configuration, meaning that it can only run over HTTPS with TLS1.2+*
```yaml
## kibana.yaml
server.protocol: http2
server.ssl.enabled: true
server.ssl.key: path/to/key
server.ssl.certificate: path/my/cerf
```
## What is this PR doing
### Add HTTP2 support for the Kibana server
#### - Plug http2 to the Kibana server
Even if HAPI was never officially updated to really support HTTP2,
node's `http`/`https`/`http2` modules are compatible enough to be able
to just instantiate an http2 server/listener and provide it to HAPI "as
a plain https listener". There were some tweaks to do (mostly silencing
a few warnings that HAPI was causing by sending http2-illegal headers
such as `Connection`), but overall, it went smoothly.
#### - Add config validation
By default, Kibana will require a valid `h2c` configuration to accept
enabling `http2`. It means that TLS must be enabled and that TLS1.2+
should at least be in the list of supported SSL protocols
(`server.ssl.supportedProtocols`). Note that default value of this
setting includes TLS1.2 and 1.3.
#### - Add escape hatch to run `h2` without `h2c`
In some situations, it may be required to enable http2 without a valid
`h2c` configuration. Kibana supports it, by setting
`server.http2.allowUnsecure` to `true`.
(*Note, however, that if http2 is enabled without TLS, ALPN protocol
negotiation won't work, meaning that most http2 agents/clients will fail
connecting unless they're explictly configured to use http2.*)
### Add documentation about this new feature
#### - Update the user-facing doc about this new `server.protocol`
setting
Update the user-facing Kibana settings documentation to include this
`http.protocol` setting (and refer to `server.http2.allowUnsecure`)
**Note: this setting, and this feature, are considered as experimental**
### Adapt our dev tooling to support running Kibana with http2 enabled
#### - Add a `--http2` flag to the dev CLI
Enabling this flag will add the proper configuration settings to run
Kibana with `http2` enabled in an (almost) valid `h2c` configutation.
*Note: when using this flag, even if listening on the same port, the
Kibana server will be accessible over https, meaning that you need to
use https in your browser to access it. Aka `http://localhost:5601`
won't work, you need to use `https://localhost:5601`. Also, we're using
the self-signed dev certificates, meaning that you must go though the
scary warning of your browser*
#### - Implement an http2-compatible base-path proxy
The current base path proxy is based on `hapi` and `hapi/h2o2`. I tried
for a bunch hours trying to hack around to make it work with http2
proxying, but ultimately gave up and implemented a new version from
scratch.
Note that with some additional efforts, this new http2 basepath proxy
could probably fully replace the existing one and be used for both http1
and http2 traffic, but it's an optimization / refactoring that did not
feel required for this PR.
### Adapt the FTR to run suites against http2
#### - Add support to run FTR test suite against an h2c-enabled Kibana
Note that with ALPN, clients using http1 should be (and are) able to
communicate with http2 Kibana, given h2c/alpn allows protocol
negitiation. So adapting our FTR tooling was not really about making it
work with http2 (which worked out of the box), but making it work with
**the self signed certifcates we use for https on dev mode**
Note that I'm not a big fan of what I had to do, however, realistically
this was the only possible approach if we want to run arbitrary test
suites with TLS/HTTP2 enabled without massively changing our FTR setup.
Operations and QA, feel free to chime in there, as this is your
territory.
#### - Change some FTR test suites to run against an HTTP2-enabled
server
I added a quick `configureHTTP2` helper function to take any "final" FTR
suite config and mutate it to enable `http2`. I then enabled it on a few
suites locally, to make sure the suites were passing correctly.
I kept two suites running with http2 enabled:
- the `console` oss functional tests
- the `home` oss functional tests
We could possibly enable it for more, but we need to figure out what
kind of strategy we want on that matter (see below)
## What is this pull request NOT doing
#### - Making sure everything works when HTTP2 is enabled
I navigated the applications quite a bit, and did not see anything
broken, however I obviously wasn't able to do a full coverage. Also, the
self-signed certificate was a huge pain to detect issues really caused
by http2 compared to issues because the local setup isn't valid `h2c`.
In theory though (famous last words) anything not doing http/1.1
specific hacks such as bfetch should work fine with http2, given that
even if using non-http2 clients, ALPN should just allow to fallback to
http/1.x (this part was tested)
#### - Enabling HTTP2 by default
PR isn't doing it for obvious reasons.
#### - Enabling HTTP2 for all FTR suites
First of all, it's not that easy, because it requires adapting various
parts of the config (and even some var env...), and we don't have any
proper way to override config "at the end". For instance, if you add the
http2 config on a top level config (e.g. the oss functional one that is
reuse by the whole world - learned the hard way), it won't work because
higher-level configs redefined (and override) the `browser` part of the
config, loosing the settings added to run the browser in insecure mode.
Secondly, I'm not sure we really need to run that many suites with http2
enabled. I learned working on that PR that we only have like one suite
where https is enabled for the Kibana server, and I feel like it could
be fine to have the same for http2. In theory it's just a protocol
change, unless parts of our apps (e.g. bfetch) are doing things that are
specific to http/1.1, switching to http2 should be an implementation
detail.
But I'd love to get @elastic/kibana-operations and @elastic/appex-qa
opinion on that one, given they have more expertise than I do on that
area.
- Running performances tests
We should absolutely run perf testing between http/1.1 over https and
http/2, to make sure that it goes into the right directly (at least in
term of user perceived speed), but I did not do it in the scope of this
PR (and @dmlemeshko is on PTO so... 😅)
## Release Note
Add support for `http2` to the Kibana server. `http2` can be enabled by
setting `server.protocol: http2` in the Kibana config file.
Note: by default, enabling `http2` requires a valid `h2c` configuration,
meaning that it can only run over HTTPS with TLS1.2+
Please refer to the Kibana config documentation for more details.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
### Summary
Removes the versioning from the _query api requests.
This ES PR removes the version from the _query requests
https://github.com/elastic/elasticsearch/pull/108919 and got backported
at 8.14 too. We need to also remove it from our side too to be in sync
with ES changes.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Connectors registered as custom integrations use the value in
`connector.serviceType` as an id. However, there are some connectors
that share a service type. This causes an error when running Kibana due
to the id clash.
This PR changes the id into a concatenation of `serviceType` and `name`
to ensure all ids are unique.
Errors before change (these no longer recur after the changes):
```log
[2024-05-28T12:06:10.514+00:00][ERROR][plugins.customIntegrations] Integration with id=confluence already exists.
[2024-05-28T12:06:10.516+00:00][ERROR][plugins.customIntegrations] Integration with id=jira already exists.
[2024-05-28T12:06:10.517+00:00][ERROR][plugins.customIntegrations] Integration with id=jira already exists.
[2024-05-28T12:06:10.518+00:00][ERROR][plugins.customIntegrations] Integration with id=salesforce already exists
```
## Summary
Removes the versioning from the _query api requests.
This ES PR removes the version from the _query requests
https://github.com/elastic/elasticsearch/pull/108919 and got backported
at 8.14 too. We need to also remove it from our side too to be in sync
with ES changes.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Related to https://github.com/elastic/kibana/issues/7104
Update supertest, superagent, and the corresponding type package, to
their latest version.
(of course, types had some signature changes and we're massively using
supertest in all our FTR suites so the whole Kibana multiverse has to
review it)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Resolves https://github.com/elastic/kibana/issues/143335.
Some history: A similar issue was reported a few years back
(https://github.com/elastic/kibana/issues/76811). The solution
(https://github.com/elastic/kibana/pull/93319) was to use the `--cache`
PEG.js [parameter](https://pegjs.org/documentation#generating-a-parser)
when generating the parser. Back when this was added, we were still
manually building the parser on demand when it was changed. Eventually
we added support for dynamically building the parser during the build
process (https://github.com/elastic/kibana/pull/145615). I'm not sure
where along the process the `cache` parameter got lost but it didn't
appear to be used when we switched.
This PR re-adds this parameter which increases performance considerably
(metrics shown in ops/sec):
```
Before using cache:
● kuery AST API › fromKueryExpression › performance › with simple expression
Received: 7110.68990544415
● kuery AST API › fromKueryExpression › performance › with complex expression
Received: 40.51361746242248
● kuery AST API › fromKueryExpression › performance › with many subqueries
Received: 17.071767133068473
After using cache:
● kuery AST API › fromKueryExpression › performance › with simple expression
Received: 8275.49109867502
● kuery AST API › fromKueryExpression › performance › with complex expression
Received: 447.0459218892934
● kuery AST API › fromKueryExpression › performance › with many subqueries
Received: 115852.43643466769
```
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
