## Summary
Some references to `elastic-images-qa` were left in the code, probably
as these pipelines were on a pending PR when the rest got changed.
Optionally, we should remove all the `imageProject` fields, and
everything we're setting defaults - it's just generating bloat.
This co-locates data fetchers with their return values. Before the all data fetching happens at once, then later joined with return values. This makes it easier to make changes and add new data fetchers.
Aggregations, operators, and commands still need to be added manually
- `::` is the only one so far landing in 8.15.0
---------
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
## Summary
This PR adds Field statistics embeddable as panel in Dashboard
By default, it will enable the ES|QL editor for the field stats panel.
It will allow for editing of the ES|QL query, and time range.
4b5438c7-051f-4627-aab1-b802c23ca652
e9bae0e4-17cf-4a86-ad70-0da9d3667b53
If and only if ES|QL is disabled, it will show the data view picker as a
fallback.
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
This PR updates the function definitions and inline docs based on the
latest metadata from Elasticsearch.
---------
Co-authored-by: Drew Tate <andrew.tate@elastic.co>
Part of https://github.com/elastic/kibana/issues/186574
## Summary
This PR migrates the Cloud Defend Plugin's route handler that consumes
`authc.getCurrentUser` to use `coreContext.security`.
Background: This PR serves as an example of a plugin migrating away from
depending on the Security plugin, which is a high priority effort for
the last release before 9.0.
### Checklist
Delete any items that are not applicable to this PR.
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Part of https://github.com/elastic/kibana/issues/186574
## Summary
This PR migrates the Alerting Plugin's server-side code that consumes
`authc.getCurrentUser` to use coreContext.security.
Background: This PR serves as an example of a plugin migrating away from
depending on the Security plugin, which is a high priority effort for
the last release before 9.0.
### Checklist
Delete any items that are not applicable to this PR.
- [X] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
This PR adds logic to server side artifact logic, in order to transform
Event Filter entries to the expected format, if process descendant
filtering is enabled.
Conditions:
- feature flag is enabled:
`xpack.securitySolution.enableExperimental.filterProcessDescendantsForEventFiltersEnabled`
- amongst the `tags`, there is the `filter_process_descendants` tag
Output: the output contains a new operator called `descendent_of`, with
all the original entries listed as the operator's values, and one
additional `event.category is process` entry.
Example input entry (only relevant fields):
```json
"entries": [
{
"field": "process.args",
"operator": "included",
"type": "match",
"value": "IGNOREMYCHILDREN"
}
],
"tags": [
"filter_process_descendants"
],
```
Example output entry:
```json
{
"entries": [
{
"type": "simple",
"entries": [
{
"operator": "included",
"type": "descendent_of", // note: writing descendEnt here to harmonise with endpoint implementation
"value": {
"entries": [
{
"type": "simple",
"entries": [
{
"field": "process.args",
"operator": "included",
"type": "exact_cased",
"value": "IGNOREMYCHILDREN"
},
{
"field": "event.category",
"operator": "included",
"type": "exact_cased",
"value": "process"
}
]
}
]
}
}
]
}
]
}
```
### e2e testing
- spin up Kibana with the FF enabled
- spin up a v15-snapshot Agent with Endpoint integration on VM, enroll
it
- create an Event Filter with the following setup, where '_Process
descendants_' is selected and `process.args` is `IGNOREMYCHILDREN`:
<img width="1454" alt="image"
src="075d300a-a353-4c8f-ab7d-08c978df0c4b">
- create a `test.sh` on your VM with the following content:
```sh
echo "Creating file test_file_$1.tmp"
touch test_file_$1.tmp
# recursively call the original script
if [ $1 -ge 1 ]
then
sh test.sh $(( $1 - 1 )) $2
fi
```
(alternatively you can use the python script from the endpoint tests,
790c611325/Python/endpoint/test/test_event_filtering.py (L439-L450))
- run the script `sh test.sh 6 DO_NOT_IGNOREMYCHILDREN`
<img width="476" alt="image"
src="283b9e9c-dd64-40f0-a107-8bb0c483058c">
- go to Security -> Explore -> Hosts to see the reported events
- I added some filters to easily see the events
<img width="632" alt="image"
src="aca83e75-fe14-424f-a514-7ef7de785f8b">
- you should see that all events are there
<img width="1236" alt="image"
src="5f4d7a74-de91-4e10-969d-2bb856c01b0a">
- now try with `sh test.sh 9 IGNOREMYCHILDREN`
<img width="429" alt="image"
src="9e476507-719e-4f52-a4f4-305259cbe674">
- now only the events of the top process are visible, the events from
descendants are not
<img width="1173" alt="image"
src="e17443ea-d1a4-4e1a-b555-11cc83313e04">
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Resolves https://github.com/elastic/response-ops-team/issues/212
## Summary
There were some serverless alerts that were triggered by projects
encountering rate limiting (SMTP 450) failures and failures due to the
recipient address being rejected (SMTP 554) when using the Cloud SMTP
preconfigured connector. Those errors are currently flagged as framework
errors by default and this PR adds code to mark these as user errors.
### Checklist
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### To verify
- You can use the cloud deploy from this PR and the following request to
test the rate limiting error (SMTP 450) and verify the `errorSource:
'user'`. The cloud email service rate limits to 500 emails per 15
minutes.
```
curl -X POST -u $USERNAME:$PASSWORD $CLOUD_URL/api/actions/connector/elastic-cloud-email/_execute \
-H 'kbn-xsrf: true' \
-H 'Content-Type: application/json' \
-d '
{
"params": {
"to": ["$YOUR_EMAIL"],
"subject": "testing email",
"message": "test"
}
}
'
```
- I haven't been able to test the SMTP 554, but it should work the same
as the SMTP 450 error.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR introduces support for notebook lists that can be set via query
parameter or via a function exported from the `search_notebooks` plugin
start object. This will be used to allow setting a different list of
notebooks for specific pages that allow the notebooks shown to be
contextual to the current user task.
This PR does not add any new lists of notebooks, as that will be done in
another PR for the static notebooks and done via the S3 file for dynamic
notebooks.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
Adds App Search nav items support to the new Search Project Navigation
Adds Workplace Search navi items support to new Search Project
Navigation
### Testing
Since the project navigation is currently gated in Kibana use the
following config settings in `kibana.dev.yml` to test and develop this
feature. All Changes should be test with this feature on and off.
```yaml
xpack.cloud.id: "ftr_fake_cloud_id:aGVsbG8uY29tOjQ0MyRFUzEyM2FiYyRrYm4xMjNhYmM="
xpack.cloud.base_url: "https://cloud.elastic.co"
xpack.cloud.deployment_url: "/deployments/deploymentId"
xpack.cloud_integrations.experiments.enabled: true
xpack.cloud_integrations.experiments.flag_overrides:
"solutionNavEnabled": true
```
The enable the search nav via a space with this dev console command:
```
POST kbn:/api/spaces/space
{
"name": "search space",
"id": "search-space",
"description": "a description",
"color": "#5c5959",
"solution": "es",
"disabledFeatures": []
}
```
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
- adds AI assistant for queries for every rule type, apart Machine
Learning
- AI assistant is shown only when query is not empty and invalid
- When user clicks on assistant it records telemetry event
`open_assistant_on_rule_query_error `
- hidden behind `AIAssistantOnRuleCreationFormEnabled` feature flag
## Design
[Design](https://www.figma.com/file/nbgUduTmTpYNXLf1vDMP7u/General-Enhancements?type=design&node-id=115%3A5166&mode=design&t=2Yi5wvS1aDoYxuyT-1)
## Demo
92435f3b-c51e-471b-940f-604a1f245e94
## Old Demoes
**Note: old demo videos use old UI design, and assistant is shown even
for valid queries.**
<details>
<summary>list of videos</summary>
### ES|QL Case 1
Simple ES|QL query validation error solving
There 2 problems in query highlighted by validation.
First, missing metadata operator
Second, operator `=` instead of `==`
By feeding query twice in Ai Assistant, I was able to get working
solution
1eb49505-b161-4fdb-ac3c-d2833c16e2cd
### ES|QL Case 2
Fixes missing _id field, when metadata operator is present
82024fcb-822e-46f1-a80a-8b9f1725816e
### EQL Case 1
fixes EQL typo
ea18ceec-92f8-4322-b359-50e689a0ef72
</details>
### Issues
Results might not be always consistent and for more complex queries they
might not correct
e3bedfd6-943c-4979-8708-f6c33d1756a6
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Fixes https://github.com/elastic/kibana/issues/183496
Fixes https://github.com/elastic/kibana/issues/180062
## Summary
Custom integration creation / integration update currently fails on
changing subobjects property on mapping with `mapper_exception` error.
This PR handles this case by triggering a rollover when the error is
received.
## Testing
- In dev tools, enter the following
```
POST logs-testintegration.test-default/_doc
{
"message": "abc"
}
```
- Navigate to `/app/observabilityOnboarding/customLogs/?category=logs` and fill it with the following:
In the screenshot the error is visible but in the current branch the request should succeed and install `testintegration` correctly.
### Checklist
- [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Follow-up of
https://github.com/elastic/kibana/pull/186304#pullrequestreview-2132968362
Improved the error messages and `Retry` button added.
### Screenshots
Before
After
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
**tl; dr;** PR removes `svlCommonPage.forceLogout();` for test suites,
that use _SAML auth_ to Kibana with
`svlCommonPage.loginWithRole(<role>)` as this call is not needed.
More details:
`svlCommonPage.forceLogout();` was introduced together with
`svlCommonPage.login()` when FTR allowed only basic authentication with
`/login` route and operator user. Mixing basic auth with Cloud SAML auth
caused flakiness, probably some of you noticed inconsistent redirects to
Cloud UI in before/after hooks. Calling `/logout` route and handling
what happens next was important to reset Kibana state.
However with `svlCommonPage.loginWithRole(<role>)` it is no longer
needed, it works this way:
1. use Cloud SAML auth (mock IDP to replicate SAML locally) to generate
session based on provided project role
2. clear browser state: delete all cookies, clear session & local
storage, verify it succeeded
3. set cookie in browser, validate role/user is applied properly by
validating user profile in browser
Since we generate only 1 session per unique role within each FTR config
run, it is not possible to have 2 different sessions for the same user.
When your next test is calling `svlCommonPage.loginWithRole(<role>)`,
the cookie in browser and browser state overall will be reset
completely.
## Summary
In this feature the following UI changes are required:
Add an accordion layout for wrap all the image link cards.
Change Layout of image cards.
Remove "Default" Header
Change second header label
<img width="1431" alt="Screenshot 2024-06-19 at 15 18 37"
src="df4b9582-d93b-45c9-89bb-3c2e52a01320">
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Angela Chuang <6295984+angorayc@users.noreply.github.com>
## Summary
Adds the licence and product type controls to the Integration assistant.
### UI
- ESS
Licence: The `Enterprise` license must be active to use the feature. It
is checked inside the _integration_assistant_ plugin itself by default
and the license paywall is displayed in case the license requirement is
not met, no need to configure anything from outside for the license
check.
- Serverless
Product type: The `Security complete` tier must be enabled to use the
feature in security projects. The _integration_assistant_ plugin exposes
a contract API to set up a custom upselling.
The product type check is performed in the
_security_solution_serverless_ plugin and the upselling component is
passed to the _integration_assistant_ plugin to be rendered.
### API
The `withAvailability` wrapper has been introduced to encapsulate this
availability check on all the routes. The `isAvailable` flag is defined
at a plugin level and passed to the router context. The flag is
defaulted to `true` and can be set to `false` by not having the
`Enterprise` license (ESS), or by calling the `setIsActive(false)`
contract exposed (serverless).
All API requests done while the license / product type requirements are
not met will be responded with:
```
404 Not Found: This API route is not available using your current license/tier.
```
### Screenshots
ESS:
Serverless:
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Link panel to go to create integrations added. It is a compressed
version of the link card displayed on the integrations page. The
integration creation feature is not enabled by default, to enable the
feature:
```
xpack.integration_assistant.enabled: true
```
When the integrationAssistant plugin is not enabled (default) the link
does not appear.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Currently, when you start both Serverless Elasticsearch (`yarn es
serverless --projectType=oblt` and Serverless Kibana (`yarn start
--serverless=oblt`) locally, by default the stack will be configured in
a way that developers are encouraged to use native users via the Kibana
native login form. The problem is that this doesn't correspond to what
our users use in production, where we solely rely on SAML-mapped users
that are different from the native users in a number of notable ways.
The problem is exacerbated by the fact that all native users you can use
during Serverless development locally **are operator users**. As a
result, this has led to a number of missed bugs in functionalities that
worked well locally but were broken for our users in production.
The only way to test Serverless Kibana locally with SAML-mapped users is
to include `--ssl` flags in both Elasticsearch and Kibana run commands,
but since it's not a default go-to command, many developers don't use
it. Also, with `--ssl` flag, both Elasticsearch and Kibana are set up
with TLS using self-signed certificates, even though TLS is only
required for Elasticsearch to configure the SAML realm. For Kibana, this
is rather annoying since browsers complain about non-trusted
certificates. Moreover, even with the `--ssl` argument, developers are
still presented with the option to pick an operator user, which has
proven to be very tempting due to a habit developed over many years.
The goal of this PR is to synchronize the local developer experience
with the experience we provide to our users in production and includes
the following changes:
* The `yarn es serverless --projectType=oblt|security|es` command will
automatically configure TLS and test-only SAML security realm for
Serverless Elasticsearch by default, but will still assume that
Serverless Kibana is NOT configured with TLS.
* The `yarn start --serverless=oblt|security|es` command will NOT
automatically configure TLS for Kibana to supress browser and HTTP
client warnings, but will still assume that Serverless Elasticsearch is
configured with TLS.
* The `yarn start --serverless=oblt|security|es` command will also try
to configure SAML authentication provider, and will hide the option to
use basic credentials to log in as an operator.
As a result, whenever a developer starts Serverless Elasticsearch and
Serverless Kibana locally with the default commands and navigates to
`http://localhost:5601` (notice `http://` and not `https://`), they will
be greeted with the following screen (only SAML project-specific roles
available by default):
__NOTE:__ If they click on `More login options` or navigate to `/login`
route directly, they will still be able to log in as an operator user,
if needed.
## Breaking changes
Unfortunately, due to the nature of Elasticsearch SAML configuration,
which requires the Kibana URL to be known at the time Elasticsearch is
started, and the fact that `kbn-es` cannot distinguish between cases
when `--ssl` is not provided and when TLS is explicitly disabled with
`--no-ssl`, the previously recommended configuration with the `--ssl`
argument for both Elasticsearch and Kibana will no longer work by
default. It will require an additional `--kibanaUrl` argument for the
`kbn es` command. For example, the following commands will lead to an
unusable experience since Elasticsearch will expect Kibana to be
configured WITHOUT TLS:
```bash
yarn es serverless --projectType=oblt|security|es --ssl
yarn start --serverless=oblt|security|es --ssl
```
This command will need to change to this:
```bash
yarn es serverless --projectType=oblt|security|es --ssl --kibanaUrl=https://localhost:5601
yarn start --serverless=oblt|security|es --ssl
```
We can potentially fix this if we somehow teach `kbn-es` CLI arguments
parser to distinguish between `--no-ssl` and missing `--ssl`, but I
didn't manage to find a way to do that.
## Supported configurations
1. Default or Elasticsearch TLS only, Elasticsearch is configured with
TLS, Kibana is configured WITHOUT TLS, SAML-test realm is configured and
native login form is hidden:
```bash
yarn es serverless --projectType=oblt|security|es [--ssl]
yarn start --serverless=oblt|security|es
```
2. Full TLS, both Elasticsearch and Kibana are configured with TLS,
SAML-test realm is configured and native login form is hidden:
```bash
yarn es serverless --projectType=oblt|security|es --ssl --kibanaUrl=https://localhost:5601
yarn start --serverless=oblt|security|es --ssl
```
3. No TLS, both Elasticsearch and Kibana are configured WITHOUT TLS,
SAML-test realm is NOT configured and only native login form for
operator users is available. We need `--no-ssl` for Kibana since by
default it assumes Elasticsearch is running with TLS enabled in
Serverless:
```bash
yarn es serverless --projectType=oblt|security|es --no-ssl
yarn start --serverless=oblt|security|es --no-ssl
```
## Notes
Forcing developers to use TLS for Elasticsearch with self-signed
certificates by default might be a little annoying for them as they
might need to adjust their tooling. If this change is merged, I'm going
to send an email with the relevant information to help developers
overcome these annoyances.
For example, if they query Elasticsearch with `curl` or `wget`, they
need to also specify `-k`/`--insecure` or `--no-check-certificate`
arguments respectively to ignore certificate errors.
TLS also makes it a bit difficult to intercept traffic coming from
Kibana to Elasticsearch with tools like Wireshark, but it's still
possible with a few more hoops. One just needs to run Kibana with a few
special command line arguments:
```bash
NODE_OPTIONS="--no-warnings --tls-cipher-list=AES128-SHA --tls-keylog=/some-path/keylog.txt" yarn start --serverless=oblt|security|es
```
And then use the created `keylog.txt` file to decrypt traffic
automatically in Wireshark: right-click an encrypted TLS packet and
click on `Protocol Preferences -> (Pre)-Master-Secret log filename…` and
choose the `keylog.txt` file in the dialog (more details can be found
[here](https://tutorialtech.net/how-to-decrypt-nodejs-tls-traffic-with-wireshark/)):
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
