# Backport
This will backport the following commits from `main` to `8.x`:
- [Additional prototype pollution protections
(#206073)](https://github.com/elastic/kibana/pull/206073)
<!--- Backport version: 9.6.4 -->
### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)
<!--BACKPORT [{"author":{"name":"Larry
Gregory","email":"larry.gregory@elastic.co"},"sourceCommit":{"committedDate":"2025-01-28T22:00:43Z","message":"Additional
prototype pollution protections (#206073)\n\n## Summary\n\n1. Extends
the server-side prototype pollution protections introduced
in\nhttps://github.com/elastic/kibana/pull/190716 to
include\n`Array.prototype`.\n2. Applies the same prototype pollution
protections to the client-side.\n\n\n### Identify risks\n\nDoes this PR
introduce any risks? For example, consider risks like hard\nto test
bugs, performance regression, potential of data loss.\n\nDescribe the
risk, its severity, and mitigation for each identified\nrisk. Invite
stakeholders and evaluate how to proceed before merging.\n\n- [ ]
Sealing prototypes on the client can lead to failures in\nthird-party
dependencies. I'm relying on sufficient functional test\ncoverage to
detect issues here. As a result, these protections are\ndisabled by
default for now, and can be controlled via
setting\n`server.prototypeHardening:
true/false`\n\n---------\n\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by:
Elastic Machine
<elasticmachine@users.noreply.github.com>","sha":"9ce2dd8df9f2bd6c0ba1d089b69ddfd7fc1f4a02","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Security","release_note:skip","Feature:Hardening","v9.0.0","backport:prev-minor","ci:cloud-deploy","ci:project-deploy-elasticsearch","ci:all-gen-ai-suites"],"title":"Additional
prototype pollution
protections","number":206073,"url":"https://github.com/elastic/kibana/pull/206073","mergeCommit":{"message":"Additional
prototype pollution protections (#206073)\n\n## Summary\n\n1. Extends
the server-side prototype pollution protections introduced
in\nhttps://github.com/elastic/kibana/pull/190716 to
include\n`Array.prototype`.\n2. Applies the same prototype pollution
protections to the client-side.\n\n\n### Identify risks\n\nDoes this PR
introduce any risks? For example, consider risks like hard\nto test
bugs, performance regression, potential of data loss.\n\nDescribe the
risk, its severity, and mitigation for each identified\nrisk. Invite
stakeholders and evaluate how to proceed before merging.\n\n- [ ]
Sealing prototypes on the client can lead to failures in\nthird-party
dependencies. I'm relying on sufficient functional test\ncoverage to
detect issues here. As a result, these protections are\ndisabled by
default for now, and can be controlled via
setting\n`server.prototypeHardening:
true/false`\n\n---------\n\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by:
Elastic Machine
<elasticmachine@users.noreply.github.com>","sha":"9ce2dd8df9f2bd6c0ba1d089b69ddfd7fc1f4a02"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/206073","number":206073,"mergeCommit":{"message":"Additional
prototype pollution protections (#206073)\n\n## Summary\n\n1. Extends
the server-side prototype pollution protections introduced
in\nhttps://github.com/elastic/kibana/pull/190716 to
include\n`Array.prototype`.\n2. Applies the same prototype pollution
protections to the client-side.\n\n\n### Identify risks\n\nDoes this PR
introduce any risks? For example, consider risks like hard\nto test
bugs, performance regression, potential of data loss.\n\nDescribe the
risk, its severity, and mitigation for each identified\nrisk. Invite
stakeholders and evaluate how to proceed before merging.\n\n- [ ]
Sealing prototypes on the client can lead to failures in\nthird-party
dependencies. I'm relying on sufficient functional test\ncoverage to
detect issues here. As a result, these protections are\ndisabled by
default for now, and can be controlled via
setting\n`server.prototypeHardening:
true/false`\n\n---------\n\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by:
Elastic Machine
<elasticmachine@users.noreply.github.com>","sha":"9ce2dd8df9f2bd6c0ba1d089b69ddfd7fc1f4a02"}}]}]
BACKPORT-->
# Backport
This will backport the following commits from `main` to `8.x`:
- [Categorise Kibana plugins - Wave 2
(#199302)](https://github.com/elastic/kibana/pull/199302)
<!--- Backport version: 9.4.3 -->
### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)
<!--BACKPORT [{"author":{"name":"Gerard
Soldevila","email":"gerard.soldevila@elastic.co"},"sourceCommit":{"committedDate":"2024-11-08T11:45:20Z","message":"Categorise
Kibana plugins - Wave 2 (#199302)\n\n## Summary\r\n\r\nThe remaining
plugins are being categorised with the\r\nsupervision/approval of their
owners.","sha":"d06856157e0d1c2e2efd561c0388287a19ed4d8f","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","release_note:skip","v9.0.0","backport:prev-minor","ci:project-deploy-observability"],"title":"Categorise
Kibana plugins - Wave
2","number":199302,"url":"https://github.com/elastic/kibana/pull/199302","mergeCommit":{"message":"Categorise
Kibana plugins - Wave 2 (#199302)\n\n## Summary\r\n\r\nThe remaining
plugins are being categorised with the\r\nsupervision/approval of their
owners.","sha":"d06856157e0d1c2e2efd561c0388287a19ed4d8f"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/199302","number":199302,"mergeCommit":{"message":"Categorise
Kibana plugins - Wave 2 (#199302)\n\n## Summary\r\n\r\nThe remaining
plugins are being categorised with the\r\nsupervision/approval of their
owners.","sha":"d06856157e0d1c2e2efd561c0388287a19ed4d8f"}}]}]
BACKPORT-->
Co-authored-by: Gerard Soldevila <gerard.soldevila@elastic.co>
## Summary
Moving common services to respective new homes.
Resolves: https://github.com/elastic/kibana/issues/188541
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
### This PR introduces a new type of API integration tests in FTR:
deployment-agnostic
#### Test suite is considered deployment-agnostic when it fulfils the
following criteria:
**Functionality**: It tests Kibana APIs that are **logically identical
in both stateful and serverless environments** for the same SAML roles.
**Design**: The test design is **clean and does not require additional
logic** to execute in either stateful or serverless environments.
### How It Works
Most existing stateful tests use basic authentication for API testing.
In contrast, serverless tests use SAML authentication with
project-specific role mapping.
Since stateful deployments also support SAML, deployment-agnostic tests
**configure Elasticsearch and Kibana with SAML authentication in both
cases**. For roles, stateful deployments define 'viewer', 'editor', and
'admin' roles with serverless-alike privileges.
New `samlAuth` service has `AuthProvider` interface with 2 different
implementations: depending on environment context (serverless or
stateful) appropriate implementation is used. But it remains on service
level and hidden in test suite.
test example
```
export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
const samlAuth = getService('samlAuth');
const supertestWithoutAuth = getService('supertestWithoutAuth');
let roleAuthc: RoleCredentials;
let internalHeaders: InternalRequestHeader;
describe('GET /api/console/api_server', () => {
before(async () => {
roleAuthc = await samlAuth.createM2mApiKeyWithRoleScope('admin');
internalHeaders = samlAuth.getInternalRequestHeader();
});
after(async () => {
await samlAuth.invalidateM2mApiKeyWithRoleScope(roleAuthc);
});
it('returns autocomplete definitions', async () => {
const { body } = await supertestWithoutAuth
.get('/api/console/api_server')
.set(roleAuthc.apiKeyHeader)
.set(internalHeaders)
.set('kbn-xsrf', 'true')
.expect(200);
expect(body.es).to.be.ok();
const {
es: { name, globals, endpoints },
} = body;
expect(name).to.be.ok();
expect(Object.keys(globals).length).to.be.above(0);
expect(Object.keys(endpoints).length).to.be.above(0);
});
});
}
```
Please read
[readme](966822ac87/x-pack/test/api_integration/deployment_agnostic/README.md)
for more details and step-by-step guide. It should help migrating
existing serverless tests to deployment-agnostic, assuming requirements
are met.
### Examples
Deployment-agnostic tests:
```
x-pack/test/api_integration/deployment_agnostic/apis/console/spec_definitions.ts
x-pack/test/api_integration/deployment_agnostic/apis/core/compression.ts
x-pack/test/api_integration/deployment_agnostic/apis/painless_lab/painless_lab.ts
```
Configs to run it:
```
node scripts/functional_tests --config x-pack/test/api_integration/deployment_agnostic/oblt.serverless.config.ts
node scripts/functional_tests --config x-pack/test/api_integration/deployment_agnostic/search.serverless.config.ts
node scripts/functional_tests --config x-pack/test/api_integration/deployment_agnostic/security.serverless.config.ts
node scripts/functional_tests --config x-pack/test/api_integration/deployment_agnostic/stateful.config.ts
```
PR is a compact version of #188737 with reduced changes in existing
serverless tests.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: elena-shostak <165678770+elena-shostak@users.noreply.github.com>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>
## Summary
1. Apply https://github.com/elastic/kibana/pull/185870 for Security
Solution Explore and Investigation tests.
2. Remove duplicated tests.
3. Investigation/timeline/serverless tests are skipped atm.
How to run the api integration tests (Use Investigation Timeline ESS as
an example)
```
cd x-pack
node scripts/functional_tests_server.js --config ./test/security_solution_api_integration/test_suites/investigation/timeline/trial_license_complete_tier/configs/ess.config.ts
// After server is started, open another terminal
cd x-pack
node ../scripts/functional_test_runner --config=test/security_solution_api_integration/test_suites/investigation/timeline/trial_license_complete_tier/configs/ess.config.ts
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Added FTR configs over http2 for security tests.
- `security_api_integration/oidc.http2.config.ts`
- `security_api_integration/saml.http2.config.ts`
- `security_functional/oidc.http2.config.ts`
- `security_functional/saml.http2.config.ts`
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
__Fixes: https://github.com/elastic/kibana/issues/184769__
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Part of https://github.com/elastic/kibana/issues/7104
Add support for `http2` to the Kibana server. `http2` can be enabled by
setting `server.protocol: http2` in the Kibana config file.
*Note: by default, enabling `http2` requires a valid `h2c`
configuration, meaning that it can only run over HTTPS with TLS1.2+*
```yaml
## kibana.yaml
server.protocol: http2
server.ssl.enabled: true
server.ssl.key: path/to/key
server.ssl.certificate: path/my/cerf
```
## What is this PR doing
### Add HTTP2 support for the Kibana server
#### - Plug http2 to the Kibana server
Even if HAPI was never officially updated to really support HTTP2,
node's `http`/`https`/`http2` modules are compatible enough to be able
to just instantiate an http2 server/listener and provide it to HAPI "as
a plain https listener". There were some tweaks to do (mostly silencing
a few warnings that HAPI was causing by sending http2-illegal headers
such as `Connection`), but overall, it went smoothly.
#### - Add config validation
By default, Kibana will require a valid `h2c` configuration to accept
enabling `http2`. It means that TLS must be enabled and that TLS1.2+
should at least be in the list of supported SSL protocols
(`server.ssl.supportedProtocols`). Note that default value of this
setting includes TLS1.2 and 1.3.
#### - Add escape hatch to run `h2` without `h2c`
In some situations, it may be required to enable http2 without a valid
`h2c` configuration. Kibana supports it, by setting
`server.http2.allowUnsecure` to `true`.
(*Note, however, that if http2 is enabled without TLS, ALPN protocol
negotiation won't work, meaning that most http2 agents/clients will fail
connecting unless they're explictly configured to use http2.*)
### Add documentation about this new feature
#### - Update the user-facing doc about this new `server.protocol`
setting
Update the user-facing Kibana settings documentation to include this
`http.protocol` setting (and refer to `server.http2.allowUnsecure`)
**Note: this setting, and this feature, are considered as experimental**
### Adapt our dev tooling to support running Kibana with http2 enabled
#### - Add a `--http2` flag to the dev CLI
Enabling this flag will add the proper configuration settings to run
Kibana with `http2` enabled in an (almost) valid `h2c` configutation.
*Note: when using this flag, even if listening on the same port, the
Kibana server will be accessible over https, meaning that you need to
use https in your browser to access it. Aka `http://localhost:5601`
won't work, you need to use `https://localhost:5601`. Also, we're using
the self-signed dev certificates, meaning that you must go though the
scary warning of your browser*
#### - Implement an http2-compatible base-path proxy
The current base path proxy is based on `hapi` and `hapi/h2o2`. I tried
for a bunch hours trying to hack around to make it work with http2
proxying, but ultimately gave up and implemented a new version from
scratch.
Note that with some additional efforts, this new http2 basepath proxy
could probably fully replace the existing one and be used for both http1
and http2 traffic, but it's an optimization / refactoring that did not
feel required for this PR.
### Adapt the FTR to run suites against http2
#### - Add support to run FTR test suite against an h2c-enabled Kibana
Note that with ALPN, clients using http1 should be (and are) able to
communicate with http2 Kibana, given h2c/alpn allows protocol
negitiation. So adapting our FTR tooling was not really about making it
work with http2 (which worked out of the box), but making it work with
**the self signed certifcates we use for https on dev mode**
Note that I'm not a big fan of what I had to do, however, realistically
this was the only possible approach if we want to run arbitrary test
suites with TLS/HTTP2 enabled without massively changing our FTR setup.
Operations and QA, feel free to chime in there, as this is your
territory.
#### - Change some FTR test suites to run against an HTTP2-enabled
server
I added a quick `configureHTTP2` helper function to take any "final" FTR
suite config and mutate it to enable `http2`. I then enabled it on a few
suites locally, to make sure the suites were passing correctly.
I kept two suites running with http2 enabled:
- the `console` oss functional tests
- the `home` oss functional tests
We could possibly enable it for more, but we need to figure out what
kind of strategy we want on that matter (see below)
## What is this pull request NOT doing
#### - Making sure everything works when HTTP2 is enabled
I navigated the applications quite a bit, and did not see anything
broken, however I obviously wasn't able to do a full coverage. Also, the
self-signed certificate was a huge pain to detect issues really caused
by http2 compared to issues because the local setup isn't valid `h2c`.
In theory though (famous last words) anything not doing http/1.1
specific hacks such as bfetch should work fine with http2, given that
even if using non-http2 clients, ALPN should just allow to fallback to
http/1.x (this part was tested)
#### - Enabling HTTP2 by default
PR isn't doing it for obvious reasons.
#### - Enabling HTTP2 for all FTR suites
First of all, it's not that easy, because it requires adapting various
parts of the config (and even some var env...), and we don't have any
proper way to override config "at the end". For instance, if you add the
http2 config on a top level config (e.g. the oss functional one that is
reuse by the whole world - learned the hard way), it won't work because
higher-level configs redefined (and override) the `browser` part of the
config, loosing the settings added to run the browser in insecure mode.
Secondly, I'm not sure we really need to run that many suites with http2
enabled. I learned working on that PR that we only have like one suite
where https is enabled for the Kibana server, and I feel like it could
be fine to have the same for http2. In theory it's just a protocol
change, unless parts of our apps (e.g. bfetch) are doing things that are
specific to http/1.1, switching to http2 should be an implementation
detail.
But I'd love to get @elastic/kibana-operations and @elastic/appex-qa
opinion on that one, given they have more expertise than I do on that
area.
- Running performances tests
We should absolutely run perf testing between http/1.1 over https and
http/2, to make sure that it goes into the right directly (at least in
term of user perceived speed), but I did not do it in the scope of this
PR (and @dmlemeshko is on PTO so... 😅)
## Release Note
Add support for `http2` to the Kibana server. `http2` can be enabled by
setting `server.protocol: http2` in the Kibana config file.
Note: by default, enabling `http2` requires a valid `h2c` configuration,
meaning that it can only run over HTTPS with TLS1.2+
Please refer to the Kibana config documentation for more details.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Related to https://github.com/elastic/kibana/issues/7104
Update supertest, superagent, and the corresponding type package, to
their latest version.
(of course, types had some signature changes and we're massively using
supertest in all our FTR suites so the whole Kibana multiverse has to
review it)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Revives this https://github.com/elastic/kibana/pull/181969
To do so, I had to create a new package `search-types` and move the
types I need there.
The Discovery team can take it from here.
Note: It also does a cleanup on the types I move, some of them were
declared twice.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Since many our e2e tests use Retry service, we might have a situation
with retry running for some time (**20,30, 120 seconds** ) before
reaching timeout while browser already crashed and Webdriver is not
functioning properly.
This PR updates Retry service with checking error name before retry
attempt: if it is the WebDriver critical error, retry is aborted and we
fail test fast.
It should help with long useless logging messages as well:
Before:
```
│ debg --- retry.try error: no such window: target window already closed
│ from unknown error: web view not found
│ (Session info: chrome=120.0.6099.129)
│ debg Find.findByCssSelector('[data-test-subj="canvasExpressionInput"]') with timeout=10000
│ debg --- retry.try failed again with the same message...
│ debg Find.findByCssSelector('[data-test-subj="canvasExpressionInput"]') with timeout=10000
│ debg --- retry.try failed again with the same message...
...
│ERROR Browser is closed, no artifacts were captured for the failure
└- ✖ fail: Canvas Canvas app expression editor shows autocomplete when typing
│ retry.try timeout: NoSuchWindowError: no such window: target window already closed
│ from unknown error: web view not found
│ (Session info: chrome=120.0.6099.129)
│ at Object.throwDecodedError (/Users/dmle/github/kibana/node_modules/selenium-webdriver/lib/error.js:524:15)
│ at parseHttpResponse (/Users/dmle/github/kibana/node_modules/selenium-webdriver/lib/http.js:601:13)
│ at Executor.execute (/Users/dmle/github/kibana/node_modules/selenium-webdriver/lib/http.js:529:28)
│ at processTicksAndRejections (node:internal/process/task_queues:95:5)
│ at Task.exec (prevent_parallel_calls.ts:28:20)
│ Error: retry.try timeout: NoSuchWindowError: no such window: target window already closed
│ from unknown error: web view not found
│ (Session info: chrome=120.0.6099.129)
│ at Object.throwDecodedError (node_modules/selenium-webdriver/lib/error.js:524:15)
│ at parseHttpResponse (node_modules/selenium-webdriver/lib/http.js:601:13)
│ at Executor.execute (node_modules/selenium-webdriver/lib/http.js:529:28)
│ at processTicksAndRejections (node:internal/process/task_queues:95:5)
│ at Task.exec (prevent_parallel_calls.ts:28:20)
│ at onFailure (retry_for_success.ts:17:9)
│ at retryForSuccess (retry_for_success.ts:59:13)
│ at RetryService.try (retry.ts:31:12)
│ at Proxy.clickByCssSelector (find.ts:417:5)
│ at TestSubjects.click (test_subjects.ts:164:5)
│ at Context.<anonymous> (expression.ts:92:7)
│ at Object.apply (wrap_function.js:73:16)
│
│
└-> "after all" hook for "shows autocomplete when typing"
│ debg unloading docs from archive at /Users/dmle/github/kibana/x-pack/test/functional/fixtures/kbn_archiver/canvas/default.json
│ info deleting 1 objects { space: undefined }
│ succ 1 saved objects deleted
└-> "after all" hook: afterTestSuite.trigger for "shows autocomplete when typing"
│ERROR Browser window is already closed
```
After:
```
│ debg --- retry.try error: no such window: target window already closed
│ from unknown error: web view not found
│ (Session info: chrome=120.0.6099.129)
│ERROR Browser is closed, no artifacts were captured for the failure
└- ✖ fail: Canvas Canvas app expression editor shows autocomplete when typing
│ Error: WebDriver session is invalid, retry was aborted
│ at retryForSuccess (retry_for_success.ts:64:13)
│ at RetryService.try (retry.ts:31:12)
│ at MonacoEditorService.getCodeEditorValue (monaco_editor.ts:25:5)
│ at Context.<anonymous> (expression.ts:83:34)
│ at Object.apply (wrap_function.js:73:16)
│
│
└-> "after all" hook for "shows autocomplete when typing"
│ debg unloading docs from archive at /Users/dmle/github/kibana/x-pack/test/functional/fixtures/kbn_archiver/canvas/default.json
│ info deleting 1 objects { space: undefined }
│ succ 1 saved objects deleted
└-> "after all" hook: afterTestSuite.trigger for "shows autocomplete when typing"
│ERROR Browser window is already closed
└-> "after all" hook in "Canvas app"
│ debg set roles = superuser
│ debg creating user test_user
│ debg created user test_user
└-> "after all" hook: afterTestSuite.trigger in "Canvas app"
│ERROR Browser window is already closed
└-> "after all" hook: afterTestSuite.trigger in "Canvas"
│ERROR Browser window is already closed
5 passing (17.0s)
1 failing
1) Canvas
Canvas app
expression editor
shows autocomplete when typing:
Error: WebDriver session is invalid, retry was aborted
at retryForSuccess (retry_for_success.ts:64:13)
at RetryService.try (retry.ts:31:12)
at MonacoEditorService.getCodeEditorValue (monaco_editor.ts:25:5)
at Context.<anonymous> (expression.ts:83:34)
at Object.apply (wrap_function.js:73:16)
```
Updates new teams as codeowners for Observability team changes.
Also took the opportunity to:
- Delete some paths that no longer exist
- Split infra code ownership between teams (from #168992)
**Addresses:** https://github.com/elastic/kibana/issues/164451
## Summary
This PR allows to run role based reused between ESS and Serverless Cypress tests.
## Details
The main idea behind is to make environmental differences for tests unnoticeable. As Serverless env already has roles and users but ESS env allows to create any possible role and user we just need to create Serverless roles and corresponding users + specific ESS roles and corresponding users in ESS env before running any ESS tests. This way tests will run in a similar env and don't have to bother by roles/users creation in test suites. This is achieved by using separate Cypress support files (Cypress includes `support/e2e.js` by default) `ess_e2e.ts` and `serverless_e2e.ts` executed for corresponding environments. `ess_e2e.ts` contains logic to create mentioned above roles and users while `serverless_e2e.ts` doesn't contain such logic.
_Only one user created per role and user has the same name as its corresponding role with `changeme` password._
To have an ability to create roles we need to store their definitions somewhere. It's also convenient to have JSON definitions instead of YAML. Plus Serverless roles should be pulled from `project-controller` repo but it's not addressed in this PR. I've chosen the following locations
- Serverless Security roles in `packages/kbn-es/src/serverless_resources/security_roles.json`. While `@kbn/es` is a common package it has `serverless_resources` folder containing `roles.yml` with a mix of `https://github.com/elastic/project-controller/blob/main/internal/project/observability/config/roles.yml`, `https://github.com/elastic/project-controller/blob/main/internal/project/esproject/config/roles.yml` and `https://github.com/elastic/project-controller/blob/main/internal/project/security/config/roles.yml` copied from `project-controller` and used for ES data restore. As there is no automation yet it looks logical to keep Security roles subset next to ES Serverless resources.
- ESS Security specific roles in `x-pack/plugins/security_solution/common/test/ess_roles.json`
On top of that the following has been done
- `reader` role replaced with `t1_analyst` where possible in tests (besides `e2e/explore/cases/attach_alert_to_case.cy.ts` but it's purely ESS test so it's fine) as `reader` is ESS specific and make harder to run the same tests in ESS and Serverless environments but both roles are almost equivalent
- `login()` helper function accepts all known roles (Serverless + ESS) but throws an exception if a custom ESS role is used under Serverless env
- `x-pack/plugins/security_solution/server/lib/detection_engine/scripts/roles_users` isn't necessary anymore as `security_roles.json` + `ess_roles.json` contain all the necessary data to create roles and users
### Does it enable role support for MKI environments?
No. This PR only enabling role support for Non-MKI Serverless environments. MKI env has predefined roles but not users. This will be addressed in a follow up PR.
## Flaky test runner
Two unskiped in this PR Serverless Cypress tests using non default role `detection_response/detection_alerts/missing_privileges_callout.cy.ts` and `detection_response/prebuilt_rules/prebuilt_rules_install_update_authorization.cy.ts` [150 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3723) 🟢 (there is one env related failure but it doesn't look related to the changes in this PR)
## Summary
This PR fixes few issues occurring while running FTR API tests against
actual serverless project.
How to run:
```
TEST_CLOUD=1 ES_SECURITY_ENABLED=1 NODE_TLS_REJECT_UNAUTHORIZED=0 TEST_ES_URL=<your_es_url_with_credentials> TEST_KIBANA_URL=<your_es_url_with_credentials> node --no-warnings scripts/functional_test_runner --es-version=8.9.0 --config x-pack/test_serverless/api_integration/test_suites/search/config.ts --bail
```
The first error is faced during Elasticsearch version validation
```
ERROR Error: attempted to use the "es" service to fetch Elasticsearch version info but the request failed: ResponseError: {"ok":false,"message":"Unknown resource."}
at SniffingTransport.request (/Users/dmle/github/kibana/node_modules/@elastic/transport/src/Transport.ts:535:17)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Client.InfoApi [as info] (/Users/dmle/github/kibana/node_modules/@elastic/elasticsearch/src/api/api/info.ts:60:10)
at FunctionalTestRunner.validateEsVersion (functional_test_runner.ts:129:16)
at functional_test_runner.ts:64:11
at FunctionalTestRunner.runHarness (functional_test_runner.ts:251:14)
at FunctionalTestRunner.run (functional_test_runner.ts:48:12)
at log.defaultLevel (cli.ts:112:32)
at run.ts:70:7
at withProcRunner (with_proc_runner.ts:29:5)
at run (run.ts:69:5)
at FunctionalTestRunner.validateEsVersion (functional_test_runner.ts:131:13)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at functional_test_runner.ts:64:11
at FunctionalTestRunner.runHarness (functional_test_runner.ts:251:14)
at FunctionalTestRunner.run (functional_test_runner.ts:48:12)
at log.defaultLevel (cli.ts:112:32)
at run.ts:70:7
at withProcRunner (with_proc_runner.ts:29:5)
at run (run.ts:69:5)
```
Since there is no version term in case of serverless, we can skip
version check by using newly added to FTR schema `serverless` property
(`false` by default). It is set to `true` in root FTR config
`/shared/config.base`.
The next error is related to ESArchiver relying on `ES` FTR service to
provide ESClient.
```
ResponseError: security_exception
│ Root causes:
│ security_exception: unable to authenticate user [system_indices_superuser] for REST request [/kibana_sample_data_flights]
```
It is fixed by using the default user (from host url) instead of
`system_indices_superuser` we use in stateful run.
## Summary
Version alllll the data view routes.
Best viewed with whitespace hidden -
https://github.com/elastic/kibana/pull/158608/files?diff=unified&w=1
In this PR:
- All REST (public and internal) routes are versioned
- Internal routes are called with version specified
- Internal and public routes are now stored in directories labeled as
such
- All routes have a response schema
- All responses are typed with `response` types, separate from internal
api types. This is to help prevent unacknowledged changes to the api.
- Moves some functional tests from js => ts
For follow up PRs:
- Move to `internal` path for internal routes
- Proper typing and schema for `fields_for_wildcard` filter
Closes https://github.com/elastic/kibana/issues/157099
Closes https://github.com/elastic/kibana/issues/157100
---------
Co-authored-by: Julia Rechkunova <julia.rechkunova@gmail.com>
## Summary
This PR removes all imports of Route from react-router-dom and
'@kbn/kibana-react-plugin/public' and instead imports Route from
@kbn/shared-ux-router.
### Context
Based on
https://github.com/elastic/kibana/issues/132629#issue-1243243678 This PR
executes steps 2 - 4:
> 2. To make the transition easier, we want to re-export other
react-router-dom exports alongside the modified' Route'.
> 3. Solutions should start using that Route component in place of the
one from react-router-dom. I.e. replace all occurrences of import { ...
} from 'react-router-dom' with import { ... } from
'@kbn/shared-ux-router'.
> 4. All manual calls to useExecutionContext are not needed anymore and
should be removed.
### Future PR
Looks like this might be getting worked on in:
https://github.com/elastic/kibana/pull/145863 (thanks!)
> Introduce an ESlint rule that ensures that react-router-dom is not
used directly in Kibana and that imports go through the new
@kbn/shared-ux-router package.
This is tangentially accomplished through
https://github.com/elastic/kibana/pull/150340 but only addresses using
Route through @kbn/kibana-react-plugin/public'
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Tiago Costa <tiagoffcc@hotmail.com>
Fixes https://github.com/elastic/kibana/issues/149344
This PR migrates all plugins to packages automatically. It does this
using `node scripts/lint_packages` to automatically migrate
`kibana.json` files to `kibana.jsonc` files. By doing this automatically
we can simplify many build and testing procedures to only support
packages, and not both "packages" and "synthetic packages" (basically
pointers to plugins).
The majority of changes are in operations related code, so we'll be
having operations review this before marking it ready for review. The
vast majority of the code owners are simply pinged because we deleted
all `kibana.json` files and replaced them with `kibana.jsonc` files, so
we plan on leaving the PR ready-for-review for about 24 hours before
merging (after feature freeze), assuming we don't have any blockers
(especially from @elastic/kibana-core since there are a few core
specific changes, though the majority were handled in #149370).
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This change ensures that the `unsafe-eval` source expression isn't included in
the Kibana Content Security Policy (CSP) by default.
Users can set `csp.disableUnsafeEval: false` to reintroduce `unsafe-eval`.
However, since this config option is deprecated as of this commit, it's
recommended to instead set `csp.script_src: ['unsafe-eval']`.
Closes#150156
The location of plugins was previously somewhat irrelevant, but as we
move into packages it's more important that we can find all plugins in
the repository, and we would like to be able to do that without needing
to maintain a manifest somewhere to accomplish this. In order to make
this possible we plan to find any plugin/package by spotting all
kibana.json files which are not "fixtures". This allows plugin-like code
(but not actual plugin code) to exist for testing purposes, but it must
be within some form of "fixtures" directory, and any plugin that isn't
in a fixtures directory will be automatically pulled into the system
(though test plugins, examples, etc. will still only be loaded when the
plugin's path is passed via `--plugin-path`, the system will know about
them and use that knowledge for other things).
Since this is just a rename Operations will review and merge by EOD Jan
12th unless someone has a blocking concern.
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* Removed comment of the issue that was referenced for the skip. But the tests were already skipped.
* Unskipping test as a fix has been made. 138510
* Made CCS test for rollups and made it conditional based on configuration.
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
* Fixed issues in build.
* Added comment to rollups test and using super user until the perms issue is fixed.
Co-authored-by: cuffs <cuffs@cuffss-Office-MacBook-Pro.local>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* [ftr] add first-class support for playwrite journeys
* [CI] Auto-commit changed files from 'node scripts/generate codeowners'
* fix jest test
* remove ability to customize kibana server args, if we need it we can add it back
* remove dev dir that doesn't exist
* fix typo
* prevent duplicated array converstion logic by sharing flag reader
* remove destructuring of option
* fix scalability config and config_path import
* fix start_servers args and tests
* include simple readme
* fix jest tests and support build re-use when changes are just to jest tests
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* add showWarning to search service
* add comments
* add unit tests
* test foo
* cleanup
* add s to property name in test
* comments for api items
* use the warnings when calling showWarnings
* change showWarning to just show a single warning
* put handleWarnings on the request adapter
* comment
* simplify 1
* fix lens unit test
* remove underscoring for unused variables
* revert inspector changes, extract the response warnings in the search service
* fix bug
* remove console.log
* re-apply typescript fixes to app test code
* declutter
* add test, improve comments
* fix some unexported public api items
* include rawResponse in the warning structure
* fix lint
* tweak clean up example app
* SearchResponseWarnings and SearchResponseWarningNotification
* fix export bug
* not include shardStats if there are no warnings
* Update src/plugins/data/common/search/types.ts
* simplify SearchResponseWarnings interface
* remove array copying
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
* comments for api_docs
* simplify per feedback
* Pass callback to handleResponse in showWarnings
* export more public types
* update example to make possible to show shard failure
* pr cleanup
* eslint fix
* allow example app to not show default warnings
* move extractWarning and related types to inspector plugin
* wip functional test of example app
* fix test references
* finish functional test
* relocate extractWarnings back to search/fetch
* fix test
* remove need for isTimeout, isShardFailure
* ts fix
* improve test
* Change showWarnings to accept the RequestAdapter
* use showWarnings in vis_types/timeseries
* more tests
* use handle_warning name
* fix ts
* add reason field to SearchResponseWarning
* fix component snapshot
* update comments
* test cleanup
* fix test
* ensure notification appears only once
* fix and cleanup
* fix ts
* fix response.json bug
* use top-level type, and lower-level reason.type
* cleanup
* fix shard failure warning in tsvb per feedback
cc @flash1293
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Joe Reuter <johannes.reuter@elastic.co>
Adds a new experimental Kibana setting called `csp.disableUnsafeEval` which will default to `false`. When set to `true`, it will remove `unsafe-eval` from our CSP.
Also introduces a new module called `@kbn/handlebars` which is a replacement for the official `handlebars` module used in the frontend. This new module is necessary in order to avoid calling `eval`/`new Function` from within `handlebars` which is not allowed once `unsafe-eval` is removed from our CSP.
The `@kbn/handlebars` module is simply an extension of the main `handlebars` module which adds a new compile function called `compileAST` (as an alternative to the regular `compile` function). This new function will not use code-generation from strings to compile the template but will instead generate an AST and return a render function with the same API as the function returned by the regular `compile` function.
This is a little bit slower method, but since this is only meant to be used client-side, the slowdown should not be an issue.
The following limitations exists when using `@kbn/handlebars`:
The Inline partials handlebars template feature is not supported.
Only the following compile options will be supported:
- `knownHelpers`
- `knownHelpersOnly`
- `strict`
- `assumeObjects`
- `noEscape`
- `data`
Only the following runtime options will be supported:
- `helpers`
- `blockParams`
- `data`
Closes#36311
This PR fixes the functional test runner for execution against cloud (and other existing deployments) by making sure the system_indices_superuser exists.
