Currently, streams logs with a `stream` field fail ingest because the
set processor tries to set a
```
"stream": {
"name": "<name field>"
}
```
which doesn't work if `"stream": "abc"` is already in the document (some
shippers do this, e.g. docker or kubernetes)
Using a painless processor this problem can be avoided and you can have
`"stream"` and `"stream.name"`
This PR addresses an issue where users with read privilege to insights
were unable to successfully complete a remediation path due to the
inability to mark an insight as remediated at the final step.
With this change, we adjust the required permissions for interacting
with the update API from writeWorkflowInsights to readWorkflowInsights.
The rationale behind this is that writeWorkflowInsights should signify
the ability to trigger new scans for insights, while
readWorkflowInsights should be sufficient for addressing found issues
without the option to generate new ones.
https://github.com/user-attachments/assets/8c1af654-d9e4-40d7-8718-1388677e8d46
Addresses #206436
- Replace styled-components with @emotion/react and @emotion/styled
across multiple components
- Remove `@kbn/ui-theme` instances
- Update test cases to use @emotion/jest matchers for toHaveStyleRule to
work properly
- Adjust styles to use `css` from `@emotion/react`
- Remove unused imports and updated snapshots
https://github.com/user-attachments/assets/48b9ab41-23fd-4c00-ace9-c0cbe58c91f7
## Summary
- Adds a new kibana feature privilege to the `Security > Endpoint`
grouping for "Manage Global Artifacts"
- new privilege is behind the following feature flag:
`endpointManagementSpaceAwarenessEnabled`
- new privilege is not yet being used in this PR. Subsequent PRs that
will introduce space awareness changes for endpoint management will
leverage this new privilege
- Fixes the `All` and `Read` labels to ensure they can be localized
(`i18n`)
## Summary
Fixes a small UI bug in the citations feature. Previously, after a
message with citations finished streaming, the citations would disappear
for a fraction of a second and then reappear again. This PR makes
improves the UI by making the citations not flash off and on after the
stream finishes.
### Changes:
- Fix flashing citations
- Refactor code related to parsing content references (to make it more
maintainable).
- Update the citations prompt slightly.
### Before:
https://github.com/user-attachments/assets/1021dd53-018a-43ba-b1f4-24aab44faca9
<img width="1782" alt="image"
src="https://github.com/user-attachments/assets/723cd29a-48a2-48e7-b031-0893484746b9"
/>
### After:
https://github.com/user-attachments/assets/21f340bc-9015-42b6-a574-0439d2f8f192
### How to test
- Enable the feature flag
```yaml
# kibana.dev.yml
xpack.securitySolution.enableExperimental: ['contentReferencesEnabled']
```
- Open the security assistant
- Ask it a question about your alerts of a document in your KB. The
response should contain citations.
- Observe the response stream carefully. Ensure the citations e.g. `[1]`
do not flash off and on when the response stream finishes. The expected
behavior is that while the message is streaming, the citations are
disabled and once the stream finishes the citations get enabled (while
always being visible).
#### Edge case to test
It is possible that citations completely disappear after streaming
finishes. This happens when the LLM produces an invalid citation.
Invalid citations are hidden client side when a message finishes
streaming. You can verify this behavior by asking GPT4o this question:
```
Prepend each line with this placeholder citation "{reference(1234)}" and append the actual citation at the end of the line. How many alerts do I have? Use the open and acknowledged alerts count tool to answer and repeat the answer 50 times on new lines.
```
While the response is getting streamed it should look like this:
<img width="200" alt="image"
src="https://github.com/user-attachments/assets/03d160bf-2404-4a4e-8701-e3183c604cc4"
/>
And when the stream finishes it should look like this:
<img width="200" alt="image"
src="https://github.com/user-attachments/assets/06367379-17da-438f-a93a-9d539067ab90"
/>
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [X] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [X]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [X] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [X] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [X] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [X] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [X] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.
Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.
- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Refactor screenshot block route to not return 404 !!
Route will return empty list instead of 404 for missing screenshot
blocks !!
This also updates to enable _inspect on route !!
### Testing
Screenshot still works as expected
<img width="1728" alt="image"
src="https://github.com/user-attachments/assets/9a2b887d-9091-4bff-97e6-3c0775e6f6bd"
/>
Unwired streams can come with or without a default ingest pipeline
defined. If there is a pipeline defined, it's clear where to plug in our
custom parsing logic.
However, so far we would bail out on streams without a configured ingest
pipeline. However, since there are good reasons to have data streams
without an ingest pipeline, we should support this case.
This PR makes sure we do:
* If the streams processing pipeline is the default pipeline itself
already, everything is fine
* If there is none yet, extend the user-managed index template to set
the default pipeline to the streams processing pipeline and roll over
the data stream to apply
## Summary
This PR deletes the "Critical Services" component from the Entity
Analytics Dashboard header.
The component was included with the service entity store, but Product
and QA consider it confusing and not very valuable to our users.
### How to test it?
* Start Kibana with security data
* Go to the security solution/entity analytics dashboard
* The component shouldn't be there
`99.1.0-borealis.0` ⏩ `99.2.0-borealis.0`
[Questions? Please see our Kibana upgrade
FAQ.](https://github.com/elastic/eui/blob/main/wiki/eui-team-processes/upgrading-kibana.md#faq-for-kibana-teams)
---
## [`v99.2.0`](https://github.com/elastic/eui/releases/v99.2.0)
- Changed `EuiFieldText` styles to prioritize `disabled` styling over
`readonly`. ([#8271](https://github.com/elastic/eui/pull/8271))
- Added `plugs` and `web` glyphs to `EuiIcon`
([#8285](https://github.com/elastic/eui/pull/8285))
- Update `title` on `EuiColorPalettePickerPaletteProps` to be optional
([#8289](https://github.com/elastic/eui/pull/8289))
**Bug fixes**
- Fixed an issue with EuiDataGrid with auto row height resulting in a
table of 0 height ([#8251](https://github.com/elastic/eui/pull/8251))
- Fixed `disabled` behavior of `EuiFieldText` to prevent input changes.
([#8271](https://github.com/elastic/eui/pull/8271))
**Borealis changes**
- [Visual Refresh] Provide hooks for color palette functions
([#8284](https://github.com/elastic/eui/pull/8284))
- [Visual Refresh] Make Borealis default theme in EUI
([#8288](https://github.com/elastic/eui/pull/8288))
---
As you can probably tell, there are plenty of updated tests and
snapshots this time around. They are the result of the following:
1. We've updated the default theme in `@elastic/eui` v99.2.0 to
Borealis, which led to kibana tests using `renderWithTheme`,
`mountWithTheme`, and other utility functions that wrap the tested
component in `<EuiThemeProvider>` to switch to the correct, new color
values. With this change, all tests should now use Borealis, regardless
of the testing framework or its configuration.
2. Because of the default theme change in EUI, some class name hash
fragments (e.g., `css-`**`wprskz`**`-euiButtonIcon-xs-empty-primary`)
regenerated.
Most Borealis-related changes in this PR are test env only and have no
impact on what users see when using Kibana.
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
- Implements #178138
- Fixes a bug where clicking on the existing link to the monitor via the
name badge led failed to load any data.
## Release Notes
- Fixes a bug where clicking on the name badge for a synthetics monitor
on an SLO details page would lead to a page that failed to load monitor
details.
- Adds a working link to the location badge on synthetics SLOs that will
route the user to the monitors page with a filter applied that matches
the location of the origin SLO.
## Summary
This PR will drastically reduce the number of console logs from the
Detection Engine in the overview cluster. If your team is depending on
the `INFO`, `WARNING`, or `verification_exception` or `ml job missing`
`ERROR` logs in that cluster, please raise your concerns here.
Addresses https://github.com/elastic/kibana-team/issues/1395,
https://github.com/elastic/kibana-team/issues/1333
Historically Detection Rules have written an enormous amount of data to
the console logs. This was helpful in debugging years ago before we had
the event log, but now we generally don't use them much. The console
logs all still get scooped up and sent to the overview cluster though.
Every rule execution writes two or more status changes (first to
'running', then to the final status later on) and these go to the
console, the event log, and the rule SO. The end result is 76% of all
logs are coming from detection rules changing status, mostly successful
statuses. These provide little value on their own.
This PR restricts console logging from detection rules to only non-user
errors. User errors and execution statuses below the error level will be
logged in the console at the `debug` level. "Unexpected" errors like
search exceptions, timeouts, etc will still appear as errors in the
console logs. The general idea is that the logs from detection rules in
the console should represent some kind of unexpected system failure.
To implement this change, I updated the console logging logic in both
the security rule execution logger. User errors reported to the
framework will still create console error logs since they're logged at
the framework level.
## Testing
Create rules that run and generate warnings (e.g. missing index), user
errors (EQL verification exceptions), and non-user errors. An easy way
to create a non-user error at the moment is running a threshold rule
when at least one index searched maps `@timestamp` as a `keyword`
instead of `date`. The non user errors still show up in console logs as
errors. User errors and warnings only show up as debug logs and with
debug logging enabled in the Kibana config.
Resolves https://github.com/elastic/kibana/issues/204467
## Summary
`assumedRequiredThroughputPerMinutePerKibana` is `NaN` when the
`capacityStats.runtime.value.load.p90` is undefined. This PR adds a
check to catch when the load.p90 is undefined, throw an error, and
ignore calculating the capacity estimation.
### Checklist
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### To verify
I was not able to reproduce this locally without changing the code, so
here is how I tested the code and I am definitely open to suggestions of
how to better test this.
1. Update the code to set `capacityStats.runtime.value.load.p90:
undefined`. I set it
[here](286c9e2ddb/x-pack/platform/plugins/shared/task_manager/server/monitoring/capacity_estimation.ts (L55)),
but there are other places upstream where you could set it to
`undefined`.
2. Start Kibana
3. Verify that you see the following log message:
```
Task manager had an issue calculating capacity estimation. averageLoadPercentage: undefined
```
## Summary
Based on a request in [this
issue](https://github.com/elastic/security-team/issues/10341). The
Cypress cleanup task for serverless testing didn't include a command to
cleanup the project tested against when that project was created using
the proxy handler. This PR adds a second conditional method in the
cleanup task to cleanup projects with the proxy handler when it is in
use.
### Checklist
- [ ] Cypress tests run with proxy handler
- [x] Projects created during a cypress run with the proxy handler are
automatically cleaned up by the cypress cleanup task
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co>
Co-authored-by: dkirchan <55240027+dkirchan@users.noreply.github.com>
Co-authored-by: Gloria Hornero <gloria.hornero@elastic.co>
## Summary
This PR fixes a regression in the `serializeState` function used to pass
the embeddable attributes to the dashboard to save the panel. In the
refactor this function started to leak the `searchSessionId` in the
serialized state and even if not used after worse it could lead to
issues and in general it's a waste of disk space.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Release notes
It fixes a regression for the serialized state for the Lens embeddable
saving attributes who should not be in the serialized state.
## Summary
This PR was initially to resolve more prompt improvements, but it will
be split into multiple PR's as it also includes a bugfix for ECS mapping
node logic, where ECS mapping node was not always part of the
translation flow.
Some minor prompt improvements are also included, an updated field
mapping for RAG rules (adding the query field) and filtering out metrics
integrations from the RAG for now.
Added telemetry metadata parameters to createModel together with
`maxRetries` as well.
Part of https://github.com/elastic/kibana/issues/204249
Clean up work from embeddable refactor. PR removes
`DashboardContainerInput` type, which is a duplicate of `DashboardState`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Resolves https://github.com/elastic/kibana/issues/208459
## Summary
The fix added in this
[PR](https://github.com/elastic/kibana/pull/206598) to call the
`mark_removed_tasks_as_unrecognized` tasks sometimes throws an error if
the task is in the middle of running. This PR adds a try/catch and a
retry to the `runSoon` call.
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/206237
Create `fleet-synced-integrations` index in Fleet setup, added async
task that populates the index with a doc that includes remote ES output
data and installed integrations data.
ES change to add `kibana_system` privileges:
https://github.com/elastic/elasticsearch/pull/121753
To test locally:
- run elasticsearch from source to apply the privilege changes, so that
`kibana_system` can create the index.
```
yarn es source -E xpack.security.authc.api_key.enabled=true -E xpack.security.authc.token.enabled=true --source-path=/Users/juliabardi/elasticsearch -E path.data=/tmp/es-data -E xpack.ml.enabled=false
```
- enable the feature flag in `kibana.dev.yml`:
`xpack.fleet.enableExperimental: ['enableSyncIntegrationsOnRemote']`
- add a remote ES output with sync enabled
- install some integrations
- wait until Fleet setup and the task runs
- verify that the index is created and contains a doc with the expected
data
```
GET fleet-synced-integrations/_search
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": "fleet-synced-integrations",
"_id": "fleet-synced-integrations",
"_score": 1,
"_source": {
"remote_es_hosts": [
{
"hosts": [
"http://remote1:80"
],
"name": "remote1",
"sync_integrations": true
}
],
"integrations": [
{
"package_version": "1.64.1",
"updated_at": "2025-02-05T11:03:02.226Z",
"package_name": "system"
}
]
}
}
]
```
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/210313
Automates the retrieval of grouping functions ( categorize and bucket )
for both the functions definitions and docs.
Buckets signatures are tricky so I overwrite them with our
implementation. Everything else is being retrieved by ES
## Release Note
Fix package name validation on Datastream page.
## Summary
Closes - #199893
If the package name starts with a number [ Only number , alphabet ,
underscore are allowed ] then some of the script processors in the
pipeline fail with dot annotation since the fields are formed like
`ctx.123_abc.something` which fails with `Illegal Argument Exception` in
script processor.
Hence the package name has additional validation on Data stream page to
restrict it to start with an alphabet or underscore instead.
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
In https://github.com/elastic/kibana/pull/208166, we added a validation
step that verifies that all zod schemas:
a) Are built with supported types, meaning types that we can analyze for
if they are applied strictly (no excess keys)
b) Are built with any side effects (refine or transform, which may
impact the type reliability)
This doesn't impact the actual parameter validation, but is instead an
additional check on the schemas we're using.
For any violation found, we log a warning, this has the downside of
increasing our total log volume and is only really needed in development
mode.
Hence, in this PR, we're only logging these warnings if Kibana is
running in dev mode, and the PR updates all consuming plugins to pass
this context along.
## Summary
This PR adds a test config category to the scout reporting. This allows
us to distinguish between UI and API FTR tests.
A new property `testConfigCategory` has been added to all FTR configs
that don't already inherit it from a higher level config.
## Summary
This PR turns the AlertsTable into a standalone component, making it
independent from the `TriggersActionsUI` plugin.
#### Removes the alerts table registry
All configuration is now managed through the AlertsTable component
props. Shared configurations are handled by giving consumers the ability
to directly provide alerts table wrapper components (see for example the
`renderAlertsTable` prop of `getCases`).
#### Moves the alerts table to dedicated package(s)
Following the feature-driven structure we're introducing for ResponseOps
(alerting) client-side packages:
- `@kbn/response-ops-alerts-table`
- `@kbn/response-ops-alerts-apis`
- `@kbn/response-ops-alerts-fields-browser`
#### Initial work on improving composition and organization
- Reorganizes the table code into a by-entity-type folder structure
(`components/`, `hooks/`, ...)
- Simplifies some components and breaks into smaller units when possible
## To verify
For consumers of the alerts table:
- Check that all your tables have the same behavior as before (columns,
sort, row actions, bulk actions, etc.)
- Check that your "shared" tables (i.e. cases alerts view in O11y and
Security) have the expected configuration and behavior
> [!WARNING]
> This PR moves a lot of files. Git might not always recognize the
correct delete/add file pairs. If you see weird diffs feel free to reach
out for help!
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
### Identify risks
| Risk | Description | Severity | Mitigation |
|---|---|---|---|
| Table misconfigurations | Some table configurations might slightly
differ from the previous AlertsTableRegistry-backed version | Low |
Quick fix |
## References
Closes#195180
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Christos Nasikas <xristosnasikas@gmail.com>
## Summary
Fix flaky tests for custom rule and step detail !!
## Changes
`custom_status_alert.journey.ts
`
Rule name filling was flaky so sometimes it was keeping the default name
value, so i have removed it to simplify and keep default rule name
always which is "Synthetics monitor status rule"
`step_details.journey.ts
`
Monitor wasn't being created due to a bad request on browser monitor
payload, which was causing this test to fail, monitor absence was
causing test to fail.
Closes https://github.com/elastic/kibana/issues/210179
## Summary
While attempting to attach an alias to the session index, we were using
`.kibana_security_session_1` as the index name. However, the session
index, if upgraded using the Upgrade assistant gets renamed to
`.kibana_security_session_1-reindexed-for-9` and
`.kibana_security_session_1` is set as an alias pointing to this index.
When we try to reattach the alias using this as the index name, ES
throws an error. This doesn't affect Kibana functionality but it
increases the number of errors thrown in the logs.
**This PR corrects this issue by attaching the alias to the index only
when alias isn't already present. We now only assign the alias if not
present - and not during creation of the index as it is created with the
alias in it's settings.**
### Release note
Fixes the assignment of the Session index alias by only attaching it if
not already present.
## How to test
To see the error in the logs, you'll need at least 7.x and 8.x checked
out locally. Once done, run bootstrap.
#### Step 1: On 7.17
- Start ES with `yarn es snapshot --license trial -E
path.data=/tmp/esdata`
- Start Kibana and login with elastic user
- You can check the contents of Kibana session index:
```
GET .kibana_security_session_1/_search
{
"query": {
"match_all": {}
}
}
```
Should return a single document
- You can now shut kibana and ES
#### Step 2: on 8.x
- Make a backup of the esdata above `cp -r /tmp/esdata /tmp/esdatabkp`
- Start ES as above `yarn es snapshot --license trial -E
path.data=/tmp/esdata`
- Start kibana and login
- Navigate to Upgrade assistant. You should see at least 2 System
indices that require migration (Security and Kibana)
- Start the migrate index process (Step 2 in the UA interface)
- Once done, trigger a local restart of kibana either by restarting
using the start script or just triggering a file save on any file in
your IDE
- You should start seeing the error described above in 100ms increments
till it reaches 10000ms and then it's every 10 seconds
- Navigate to Dev tools and run
```
GET .kibana_security_session/_alias
```
You should see the index as
```
.kibana_security_session_reindexed-for-9 {
aliases: {
// aliases of the index including kibana_security_session_1
}
}
```
#### Verify the fix
To verify, we have a couple of options - either clone the PR and go
through the same steps as Step 2 above but for 9.0. The easier option is
replace the code of function `attachAliasToIndex` in `session_index.ts`
in 8.x with the changes in this PR. This should restart your kibana
server and you will no longer see the error in the logs.
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
While working on improving the history feature of the expandable flyout
package, I realized that we've kept a `state` property on. our
`flyoutPanelProps` interface that has been unused since the flyout got
released. This property was intended to be used for some scenario we had
imagined, but over the last couple of years of using the expandable
flyout, it was never needed.
This PR makes a very small cleanup to remove that `state` property. We
can add it back on day if a need arises.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
- [x] Removed use of color calculating function `transparentize()`
- [x] Begin removal of JSON tokens when possible in favor of colors
tokens
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
