EPIC: https://github.com/elastic/kibana/issues/144943
## Summary
Update Events/alerts table to provide `CellActions` with a complete
`FieldSpec`object from DataView
### Affected pages:
* Alerts page
* Security Dashboards
* Rule preview
* Host events
* Users events
### How to test it
Use CellActions on one of the affected pages.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
relates to [#160381](https://github.com/elastic/kibana/issues/160381)
closes [#161432](https://github.com/elastic/kibana/issues/161432)
## Summary
This pull request aims to enhance the usage of Lens Embeddable by
introducing abstractions that simplify the creation of the necessary
data structures for rendering charts. The goal is to improve the DX by
providing clearer and more intuitive interfaces.
More details in the
[README.md](62a9ef70e6/x-pack/plugins/infra/public/common/visualizations/lens/README.md)
### For reviewers
- The majority of the changes are concentrated in the
`common/visualizations` directory. The `formulas/host` files have been
modified to only contain the formula itself. The formulas are now
designed to be independent of specific chart styles.
- The `use_lens_attributes` hook has been modified to receive the
essential information. Previously, it had knowledge of host formulas,
but now it accepts an abstracted chart representation and returns the
JSON object accordingly.
- Chart-specific style configurations have been moved to the chart usage
component in the `tile` and `metric_chart` components.
### How to test
- Start a local Kibana instance
- Navigate to `Infrastructure` > `Hosts`
- Check all charts and navigate to Lens.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Sets up unit testing for react components.
Unit testing of server-side components already implemented.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
Closes https://github.com/elastic/kibana/issues/159558
### What was done
Added feature flag in deep linking to hide Storage Explorer from the
global search in serverless
### Before
<img width="1113" alt="Screenshot 2023-07-04 at 09 25 11"
src="6f4ecfad-8067-4c40-8478-15550463f3d0">
### After
<img width="1113" alt="Screenshot 2023-07-04 at 09 26 13"
src="2d33214b-4d30-4836-a6b2-c6878ef1ba29">
Adds checks to avoid unnecessary initial rerenders and refetches of data
when loading the transform wizard. This reduces rerenders from 14 to 11
and more importantly the request to fetch index data is triggered only
once instead of three times. One of the requests even triggered an error
which could briefly trigger a callout on slower connections.
## Summary
This PR adds UI validation for comments maximum length.
It shows error message and disables save button when the comment exceeds
30k characters while
- **Adding a new comment**
- **Updating an existing comment**
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### For maintainers
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR fixes few issues occurring while running FTR API tests against
actual serverless project.
How to run:
```
TEST_CLOUD=1 ES_SECURITY_ENABLED=1 NODE_TLS_REJECT_UNAUTHORIZED=0 TEST_ES_URL=<your_es_url_with_credentials> TEST_KIBANA_URL=<your_es_url_with_credentials> node --no-warnings scripts/functional_test_runner --es-version=8.9.0 --config x-pack/test_serverless/api_integration/test_suites/search/config.ts --bail
```
The first error is faced during Elasticsearch version validation
```
ERROR Error: attempted to use the "es" service to fetch Elasticsearch version info but the request failed: ResponseError: {"ok":false,"message":"Unknown resource."}
at SniffingTransport.request (/Users/dmle/github/kibana/node_modules/@elastic/transport/src/Transport.ts:535:17)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Client.InfoApi [as info] (/Users/dmle/github/kibana/node_modules/@elastic/elasticsearch/src/api/api/info.ts:60:10)
at FunctionalTestRunner.validateEsVersion (functional_test_runner.ts:129:16)
at functional_test_runner.ts:64:11
at FunctionalTestRunner.runHarness (functional_test_runner.ts:251:14)
at FunctionalTestRunner.run (functional_test_runner.ts:48:12)
at log.defaultLevel (cli.ts:112:32)
at run.ts:70:7
at withProcRunner (with_proc_runner.ts:29:5)
at run (run.ts:69:5)
at FunctionalTestRunner.validateEsVersion (functional_test_runner.ts:131:13)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at functional_test_runner.ts:64:11
at FunctionalTestRunner.runHarness (functional_test_runner.ts:251:14)
at FunctionalTestRunner.run (functional_test_runner.ts:48:12)
at log.defaultLevel (cli.ts:112:32)
at run.ts:70:7
at withProcRunner (with_proc_runner.ts:29:5)
at run (run.ts:69:5)
```
Since there is no version term in case of serverless, we can skip
version check by using newly added to FTR schema `serverless` property
(`false` by default). It is set to `true` in root FTR config
`/shared/config.base`.
The next error is related to ESArchiver relying on `ES` FTR service to
provide ESClient.
```
ResponseError: security_exception
│ Root causes:
│ security_exception: unable to authenticate user [system_indices_superuser] for REST request [/kibana_sample_data_flights]
```
It is fixed by using the default user (from host url) instead of
`system_indices_superuser` we use in stateful run.
## Summary
Attaching ELSER pipelines to an index was disabled when ELSER pipelines
were introduced. This PR enables this feature with the following logic:
- An ELSER pipeline is selectable if all the source fields in its
`fieldMapping` configuration are present in the index. Otherwise it's
disabled in the dropdown with a message indicating which fields are
missing.
- When an ELSER pipeline is selected for attachment to an index, the
field configuration screen shows a read-only version of the field
mappings. The modification widgets (field selector dropdown, Add button,
delete button in the list) are hidden in this mode.
ELSER pipeline is selectable:
ELSER pipeline is not selectable if source fields are missing from the
index mapping:
Field configuration panel in read-only mode:
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [x] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR:
- Adds the ability to create system action types
- Creates system connectors on Kibana `start` from the system action
types
- Prevents system action to be created/updated/deleted
- Return system actions from the get/getAll endpoints
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### For maintainers
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Some refactor broke our labels, added a test case to make sure it's part
of e2e tests.
### After
<img width="1456" alt="image"
src="23f8cd56-ad4b-4c9d-97e7-2681c963406d">
### Before
<img width="1495" alt="image"
src="f835742b-d506-4beb-b68f-1d51d164ad96">
## Summary
This PR removes `onClose` prop from the expandable flyout as it is not
necessary anymore
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Closes
#155723,#155846,#156045,#156041,#156074,#156130,#156151,#156739,#157673,#156422,#156998,#156922,#156921,#157452,#158366,#158586,#158711
## Summary
Issue was resolved by PR #160584. This PR unskips saved object tests
documented as skipped in #158918.
### Testing
Flaky test runner:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2543
Closes #160376
## Summary
This PR adds the initial version of the Overview tab including KPI tiles
and metadata summary.
<img width="865" alt="image"
src="36b62899-64e8-4359-a7f5-e09a2a300bba">
The storybook is not showing the lens charts as it will be hard because
of the dependencies (we should consider if it is worth the effort to add
them there in the future) The other parts look Ok:
<img width="2550" alt="image"
src="348b868b-9ed9-4b59-b02c-cb1e91eecb5a">
## Next steps 👣
There are still some parts that can be addressed separately when other
parts are ready:
- The date is currently relying on the host date picker (passed as an
override) which can be fixed once the date picker is added to the page
## Testing
- Go to host view and open the flyout - the default tab is now overview
tab
- Hint: to see the chart movements better pick a wider time range (5
days for example)
<img width="1468" alt="image"
src="d31faf71-5beb-48f2-b20a-7dfcacd52325">
- Show all button in the metadata section should open the metadata tab
- Storybook: Use `yarn storybook infra` to run the storybook and check
both `Page` and `Flyout`
## Summary
Addresses root cause of https://github.com/elastic/kibana/issues/158918
Underlying cause is that _esArchiver_ is messing up with the SO indices
whilst Kibana is already running.
This can cause some asynchronous calls made by Kibana (e.g. `GET
/.kibana_8.8.0/telemetry:telemetry`) to hit ES at the exact time where
the underlying SO indices are **just** recreated, causing the error
described in the related issue.
The idea of the fix is to delete `mappings.json`, used by _esArchiver_
to create the SO indices. This way, _esArchiver_ will use existing SO
indices instead (aka the "official" ones, created by Kibana at startup),
thus avoiding the problem altogether.
As a side effect:
- Documents in `data.json` must be updated so that they are correctly
inserted.
- The different FTR tests must make sure the SO indices are empty before
inserting those documents (done in the `before(), beforeEach()`
statements).
Connected to https://github.com/elastic/kibana/issues/146945
## Summary
| Description | Limit | Done? | Documented?
| ------------- | ---- | :---: | ---- |
| Total number of attachments that can be created using the bulk create
API | 100 | ✅ | No |
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
@gchaps and I met to review text on some APM UI pages. Outcomes:
1. Service **m**ap or Service **M**ap — there is inconsistency in the
APM UI with how we refer to Service maps. In some cases, we use title
case (Service Map). In others, we use sentence case (Service map). As
per the [EUI writing
guidelines](https://eui.elastic.co/#/guidelines/writing/guidelines#capitalization),
we should use title case for product features.
2. Storage **e**xplorer or Storage **E**xplorer — same story here. We
use title case sometimes and sentence case others. We should use title
case as this is a product feature.
3. Various text enhancements and changes.
Fixes: https://github.com/elastic/kibana/issues/161305
## Summary
- Passes a new `immutable` params to the `upgradeRule` method that is
used when upgrading rules.
- Looks like we had a longstanding bug here in which rule updates of
rule types that changed the type of the rule were overwriting the
`immutable` prop to `false`. (Actually, those rules were deleted and
recreated with `immutable: false`)
- This was causing the `fetchAllInstalledRules` method of our
`ruleObjectsClient` NOT to retrieve these rules when they were already
installed.
- Since our installation `_review` and `_perform` endpoint depends on
this client, these rules that had had their types updated were being
incorrectly listed as available for installation.
## Testing
Repeat testing steps laid out in:
https://github.com/elastic/kibana/issues/161305
Rules shouldn't be duplicated.
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Summary
Fix an issue found in 8.8.2 with `8.7.1` inactive agents. The agents
list endpoint was returning an exception, after some investigation it
was found that the issue was a missing `units` property in the
components field of the agents.
This was a very specific edge case and in fact I could not reproduce
locally, but it still happened in a minority of cases after upgrading to
8.8.2.
The agents in this case have components that look like this:
```
"components": [
{
"id": "http/metrics-monitoring",
"type": "http/metrics",
"message": "Starting",
"status": "STARTING"
},
...
]
```
But this property was typed as mandatory so we didn't check for
undefined. The actual bug was tracked here:
2070836060/x-pack/plugins/fleet/server/services/agents/helpers.ts (L50)
and was causing this error:
607b5aed-155b-4509-896a-ada7507e4dc3
I also updated the type to have `units` as optional and fixed a couple
of other places where this could cause issues.
### Checklist
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
The previous description
> Enables the comparison feature in the APM app.
makes it sound like the entire feature would be disabled when setting
the option to false.
## Summary
This PR handles bugs
- elastic/security-team#6977
- https://github.com/elastic/security-team/issues/6978
- elastic/security-team#6979.
Currently, below operations between System Prompts and Conversarions do
not work.
1. When a prompt is set as default for all conversation, it should be
automatically selected for any new conversation user creates.
2. When a new prompt is creates and set as default for all conversation,
it should be automatically selected for any new conversation user
creates.
3. When a prompt is edited such that, it is default for only certain
conversation, it should be automatically selected for that conversation.
4. When a prompt is edited such that conversations are removed to have
that default prompt, it should be automatically removed from
conversation default system prompt list.
In addition to above scenarios, this PR also handles one more bug.
Consider below interface of Conversation which has a property
`apiConfig.defaultSystemPrompt` is of type Prompt. It has been changed
from `defaultSystemPrompt?: Prompt` to `defaultSystemPrompt?: string`
where it will store `promptId` instead of complete prompt.
The current model was posing a problem where, if a prompt was updated,
all its copies in `Conversation` were needed to be updated leading to
inconsistencies. This is now resolved.
```typescript
export interface Conversation {
apiConfig: {
connectorId?: string;
defaultSystemPrompt?: Prompt;
provider?: OpenAiProviderType;
};
id: string;
messages: Message[];
replacements?: Record<string, string>;
theme?: ConversationTheme;
isDefault?: boolean;
}
```
## Summary
Closes https://github.com/elastic/kibana/issues/160052
This PR adds support for the header banner in serverless projects.
<img width="1437" alt="image"
src="c950bbe7-46bb-4b82-b9ee-406aef7748dc">
**NOTE:** This PR adds support for the header banner by applying the
same techniques as in the "classic" layout: special class names of
`header__bar`/`header_firstBar`/`header__secondBar`. The effect is the
"header action menu" bar, which is an optional 2nd bar in the chrome
header, works better after this change when it is always visible. This
is a temporary issue that unblocks the header banner, and fixes other
overlapping issues with the current project layout stylings.
_As followup work_, the @elastic/appex-sharedux team will investigate
converting the Kibana page template to the new
[EuiPageTemplate](https://elastic.github.io/eui/#/templates/page-template/examples)
which supports horizontal sections of a page that have dynamic height --
we need this for the dynamic presence of the header banner, and the
intended dynamic presence of the header action menu.
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [x] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [x] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [x] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
Fixes#116128
# Summary
This PR changes how the CPU Usage Rule calculates the usage percentage
for containerized clusters.
Based on the comment
[here](https://github.com/elastic/kibana/issues/116128#issuecomment-1117578705),
my understanding of the issue was that because we were using a
`date_histogram` to grab the values we could sometimes run into issues
around how `date_histogram` rounds the time range and aligns it towards
the start rather than the end, causing the last bucket to be incomplete,
this is aggravated by the fact that we make the fixed duration of the
histogram the size of the lookback window.
I took a slightly different path for the rewrite, rather than using the
derivative I just look at the usage across the whole range using a
simple delta.
This has a glaring flaw in that it cannot account for the limits
changing within the lookback window (going higher/lower or set/unset),
which we will have to try to address in
https://github.com/elastic/kibana/issues/160905. The changes in this PR
should make the situation better in the other cases and it makes clear
when the limits have changed by firing alerts.
https://github.com/elastic/kibana/issues/160897 outlines follow up work
to align how the CPU usage is presented in other places in the UI.
# Screenshots
**Above threshold:**
<img width="1331" alt="above-threshold"
src="4dc4dc2a-a858-4022-8407-8179ec3115df">
**Failed to compute usage:**
<img width="1324" alt="failed-to-compute"
src="88cb3794-6466-4881-acea-002a4f81c34e">
**Limits changed:**
<img width="2082" alt="limits-changed"
src="d0526421-9362-4695-ab00-af69aa9838c9">
**Limits missing:**
<img width="1743" alt="missing-resource-limits"
src="82626968-8b18-453d-9cf8-8a6776a6a46e">
**Unexpected limits:**
<img width="1637" alt="unexpected-resource-limits"
src="721deb15-d75b-4915-8f77-b18d0b33da7d">
# CPU usage for the Completely Fair Scheduler (CFS) for Control Groups
(cgroup)
The way CPU usage for containers is calculated is this formula:
`execution_time / (time_quota_per_schedule_period * number_of_periods)`
Execution time is a counter of how many cycles the container was allowed
to execute for by the scheduler, the quota is the limit of how many
cycles are allowed per period.
The number of periods is derived from the length of the period which can
also be changed. the default being 0.1 seconds.
At the end of each period, the available cycles is refilled to
`time_quota_per_schedule_period`. With a longer period, you're likely to
be throttled more often since you'll have to wait longer for a refresh,
so once you've used your allowance for that period you're blocked. With
a shorter period you're getting refilled more often so your total
available usage is higher.
Both scenarios have an effect on your percentage CPU usage but the
number of elapsed periods is a proxy for both of these cases. If you
wanted to know about throttling compared to only CPU usage then you
might want a separate rule for that stat. In short, 100% CPU usage means
you're being throttled to some degree. The number of periods is a safe
proxy for the details of period length as the period length will only
affect the rate at which quota is refreshed.
These fields are counters, so for any given time range, we need to grab
the biggest value (the latest) and subtract from that the lowest value
(the earliest) to get the delta, then we plug those delta values into
the formula above to get the factor (then multiply by 100 to make that a
percentage). The code also has some unit conversion because the quota is
in microseconds while the usage is in nano seconds.
# How to test
There are 3 main states to test:
No limit set but Kibana configured to use container stats.
Limit changed during lookback period (to/from real value, to/from no
limit).
Limit set and CPU usage crossing threshold and then falling down to
recovery
**Note: Please also test the non-container use case for this rule to
ensure that didn't get broken during this refactor**
**1. Start Elasticsearch in a container without setting the CPU
limits:**
```
docker network create elastic
docker run --name es01 --net elastic -p 9201:9200 -e xpack.license.self_generated.type=trial -it docker.elastic.co/elasticsearch/elasticsearch:master-SNAPSHOT
```
(We're using `master-SNAPSHOT` to include a recent fix to reporting for
cgroup v2)
Make note of the generated password for the `elastic` user.
**2. Start another Elasticsearch instance to act as the monitoring
cluster**
**3. Configure Kibana to connect to the monitoring cluster and start
it**
**4. Configure Metricbeat to collect metrics from the Docker cluster and
ship them to the monitoring cluster, then start it**
Execute the below command next to the Metricbeat binary to grab the CA
certificate from the Elasticsearch cluster.
```
docker cp es01:/usr/share/elasticsearch/config/certs/http_ca.crt .
```
Use the `elastic` password and the CA certificate to configure the
`elasticsearch` module:
```
- module: elasticsearch
xpack.enabled: true
period: 10s
hosts:
- "https://localhost:9201"
username: "elastic"
password: "PASSWORD"
ssl.certificate_authorities: "PATH_TO_CERT/http_ca.crt"
```
**5. Configure an alert in Kibana with a chosen threshold**
OBSERVE: Alert gets fired to inform you that there looks to be a
misconfiguration, together with reporting the current value for the
fallback metric (warning if the fallback metric is below threshold,
danger is if is above).
**6. Set limit**
First stop ES using `docker stop es01`, then set the limit using `docker
update --cpus=1 es01` and start it again using `docker start es01`.
After a brief delay you should now see the alert change to a warning
about the limits having changed during the alert lookback period and
stating that the CPU usage could not be confidently calculated.
Wait for change event to pass out of lookback window.
**7. Generate load on the monitored cluster**
[Slingshot](https://github.com/elastic/slingshot) is an option. After
you clone it, you need to update the `package.json` to match [this
change](8bfa8351de/package.json (L45-L46))
before running `npm install`.
Then you can modify the value for `elasticsearch` in the
`configs/hosts.json` file like this:
```
"elasticsearch": {
"node": "https://localhost:9201",
"auth": {
"username": "elastic",
"password": "PASSWORD"
},
"ssl": {
"ca": "PATH_TO_CERT/http_ca.crt",
"rejectUnauthorized": false
}
}
```
Then you can start one or more instances of Slingshot like this:
`npx ts-node bin/slingshot load --config configs/hosts.json`
**7. Observe the alert firing in the logs**
Assuming you're using a connector for server log output, you should see
a message like below once the threshold is breached:
```
`[2023-06-13T13:05:50.036+02:00][INFO ][plugins.actions.server-log] Server log: CPU usage alert is firing for node e76ce10526e2 in cluster: docker-cluster. [View node](/app/monitoring#/elasticsearch/nodes/OyDWTz1PS-aEwjqcPN2vNQ?_g=(cluster_uuid:kasJK8VyTG6xNZ2PFPAtYg))`
```
The alert should also be visible in the Stack Monitoring UI overview
page.
At this point you can stop Slingshot and confirm that the alert recovers
once CPU usage goes back down below the threshold.
**8. Stop the load and confirm that the rule recovers.**
# A second opinion
I made a little dashboard to replicate what the graph in SM and the rule
**_should_** see:
[cpu_usage_dashboard.ndjson.zip](11728315/cpu_usage_dashboard.ndjson.zip)
If you want to play with the data, I've collected an `es_archive` which
you can load like this:
`node scripts/es_archiver load PATH_TO_ARCHIVE/containerized_cpu_load
--es-url http://elastic:changeme@localhost:9200 --kibana-url
http://elastic:changeme@localhost:5601/__UNSAFE_bypassBasePath`
[containerized_cpu_load.zip](11754646/containerized_cpu_load.zip)
These are the timestamps to view the data:
Start: Jun 13, 2023 @ 11:40:00.000
End: Jun 13, 2023 @ 12:40:00.000
CPU average: 52.76%
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
