## Summary
Resolves https://github.com/elastic/kibana/issues/218843
This PR removes the final reference of `REMOVED_TYPES` that was in V2
migration algorithm. Now removed types is retrieved from the
typeRegistry as it should since
https://github.com/elastic/kibana/pull/218139. Thanks to this we can now
move `REMOVED_TYPES` to `core-saved-objects-server-internal` because the
circular dependency is not there anymore.
The ticket also mentioned:
```
We can also take advantage of this task and remove excludeOnUpgradeQuery ... We can remove it and create it when needed using the type registry legacy types, this way we don't need to carry that query string in all states.
```
But when doing this I noticed that this value is actually being updated
inside the migration so decided not to change it for now.
7220f7a3ef/src/core/packages/saved-objects/migration-server-internal/src/model/model.ts (L737-L754)
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
- Cleans up the `parser.ts` module, where ES|QL ANTLR parser is
constructed.
- Create a distinct `Parser` class, which will hold all parsing related
logic.
- Removes some unnecessary imports and deprecates more imports.
- ANTLR parser and ANTLR lexer are now internal
- This sets up for removal of the `ESQLAstBuilderListener` class, which
seems to be completely unnecessary and actually counterproductive in our
parser. ANTLR has two CST traversal methods (listener and manual), this
listener class is used only to traverse the top level (commands), but
most of our parsing is then done using manual traversal. It is
inconsistent and actually bad, because this way we cannot support nested
sub-queries.
## New API
Create a parser:
```ts
const parser = Parser.create(src);
```
Parse ES|QL AST from src:
```ts
const { root } = Parser.parse(src);
```
Get parsing errors only:
```ts
const errors = Parser.parseErrors(src);
```
## Summary
Implements different behaviour for unavailable vs unauthorised links.
### Challenge
In serverless, the application links implementation did not discriminate
between these 2 scenarios:
- **unavailable**: The link can not be accessed because it's not
available in the current payment plan (aka PLI scenario)
- **unauthorized**: The link can not be accessed because the user does
not have sufficient privileges (aka RBAC scenario)
This happened because, in serverless, both scenarios are checked via
`capabilities`, the behaviour for when the capabilities check did not
pass was delegated to the components, so we were always doing the same
fallback (show no privilege page or redirect to landing) for both
scenarios.
This is a problem for links that need to do different things in each
scenario.
### Proposal
The capability check is split to discriminate between both scenarios,
and a new computed property is introduced: `unavailable`.
Example with `securitySolutionAttackDiscovery` feature:
- When all conditions are met, PLI is enabled, and the user has the
right privileges, the capabilities are `true`:
- When we are in the RBAC scenario, where the user role does not have
the required privileges, the relevant capabilities are `false`:
- In the disabled PLI scenario, the capabilities do not even exist,
because they were not registered in the Kibana feature privileges:
We can distinguish between these scenarios and act accordingly,
consistently across the app, without relying on each route component
decision (`redirectOnMissing` prop).
### Scenarios
- **Available and authorized**:
- left nav: shown
- global search: shown
- content: the page
Serverless
Classic
- **Unauthorized**:
- left nav: hidden
- global search: hidden
- content: the generic _NoPrivilege_ page. Before this PR, we were
sometimes redirecting to the landing page (when using the
`redirectOnMissing` flag).
Serverless
Classic
- **Unavailable**: When the links needs a higher the payment plan
- **With Upselling**: (same behaviour)
- left nav: shown
- global search: hidden
- content: the registered upselling component
Serverless
Classic
- **Without Upselling**:
- left nav: hidden
- global search: hidden
- content: redirects to the landing page. Before this PR, we were
sometimes showing the _NoPrivilege_ page (when the `redirectOnMissing`
flag was missing).
Serverless
Classic
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Modified the /stream route and redirect handlers from Logs Explorer to
carry state parameters when redirecting between:
Logs Stream → Logs Explorer
Logs Stream → Discover
Logs Explorer → Discover
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
When using `theme$` with
[`useObservable`](ad33f76dff/src/useObservable.ts (L10-L21))
passing custom default `theme`, the resulting `theme` can be wrong.
Take for example this code...
```ts
import useObservable from 'react-use/lib/useObservable';
const theme = useObservable<CoreTheme>(kibanaTheme.theme$, {
darkMode: false,
name: 'amsterdam',
});
```
In such case `kibanaTheme.theme$` has the correct value but the
`useObservable` returns the default/initial value immediately, so the
default is always applied then updated, requiring 2 renderings just to
update to the correct theme.
The simplest approach to fix this is just to pass the
`kibanaTheme.getTheme()` as the default when using with `useObservable`.
```ts
const theme = useObservable<CoreTheme>(kibanaTheme.theme$, kibanaTheme.getTheme());
```
---
Ideally, in the future we have a commonly shared way to access the theme
in react via a `useKibanaTheme` hook or a better/more consistent API for
`useEuiTheme`.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Edgar Santos <edgar.santos@elastic.co>
## Summary
This moves shared test utilities to a separate file, because importing
them to another test file caused both files' tests to be run there. It
looks like there are a few more instances of this throughout kibana, but
for now this is the only one under our ownership.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
Resolves https://github.com/elastic/security-team/issues/12483
This PR changes REST API Endpoints scheme to align with
https://github.com/elastic/security-team/issues/12483. Below is the
summary of changes done.
### API Scheme changes
The REST API scheme has been changed to reflect
https://github.com/elastic/security-team/issues/12483. This is pretty
much self explanatory as defined in below openapi schema yaml :
-
[x-pack/solutions/security/plugins/security_solution/common/siem_migrations/model/api/rules/rule_migration.schema.yaml](https://github.com/elastic/kibana/pull/219597/files#diff-3025af9eca156f3308474e2b42808da1531423457b7791daf6660db95a53b978)
### Introduction of Delete Migration API
This PR also adds `DELETE` method on route
`/rules/siem_migrations/{migration_id}` for deleting a migrations.
Deleting a migration does below operations:
- Stops a migration if it is running
- Deletes the rules, resources related to migration and migration
document itself.
### File Reorganizations
Directly structure has been changed a little bit to reflect the
endpoint. There is a sub-directory called `rules` which deals with only
`rules` of the migration and the root directly only contains the
endpoints related to the migration.
#### Before
```
//siem_migrations/rules/api
├── create.ts
├── get.ts
├── start.ts
├── update.ts
├──
├──
├──
```
#### After
```
//siem_migrations/rules/api
├── create.ts
├── delete.ts
├── get.ts
├── rules
│ ├── add.ts
│ ├── get.ts
│ └── update.ts
├──
├──
├──
```
## Migration Strategy
### TL,DR;
```mermaid
flowchart TD
StartM[Start Migration] --> isMigExists{Does Migration Index Exists}
isMigExists -->|Yes|FetchMDoc[Fetch Migration Docs]
isMigExists -->|No|CreateMIndex[Create MigrationIndex]
CreateMIndex --> FetchMDoc
FetchMDoc --> FetchMRules[Fetch Migration Stats Rules index]
FetchMRules --> FilterMigration{Filter Migration Docs not in Migration Index}
FilterMigration --> |is Empty|END[END]
FilterMigration --> |is Not Empty| CreateMDocs[Create Migration Docs]
CreateMDocs --> END
```
At the time of merging this PR, the Migration indices can be in 3
states:
### There are migrations created after
https://github.com/elastic/kibana/pull/216164 and this means that there
are `some` migrations existing in
`.kibana-siem-migrations-migrations-<space_id>` and migrations created
before above mentioned PR will only exist in
`.kibana-siem-migrations-rules-<space_id>`.
In this case `migrateRuleMigrationIndex` will create migration in below
steps:
1. Look for **all** migration Documents in
`.kibana-siem-migrations-migrations-<space_id>`
2. Get **all** Migrations stats from
`.kibana-siem-migrations-rules-<space_id>` which includes below
properties
- migration_id : will help in reconciling the migration id in
.kibana-siem-migrations-migrations-<space_id>` index
- created_at : Date on which migration_id was created.
- created_by: User who created the migrations.
3. A new document with above migration will be added to
`.kibana-siem-migrations-migrations-<space_id>`.
4. Now both `.kibana-siem-migrations-migrations-<space_id>` and
`.kibana-siem-migrations-rules-<space_id>` will be in sync.
### Alternatively, there are no migration created after
https://github.com/elastic/kibana/pull/216164. In that case, there is a
possibility that `.kibana-siem-migrations-migrations-<space_id>`, will
not even exist.
In this case `migrateRuleMigrationIndex` will create migration in below
steps:
1. Create the `.kibana-siem-migrations-migrations-<space_id>` index.
2. Do steps mentioned in above scenario.
### Once the migrations has been run successfully, both
`.kibana-siem-migrations-migrations-<space_id>` index and
`.kibana-siem-migrations-rules-<space_id>` will be in sync.
1. In this case, migration will not run, since it tries to filter the
migrations by `id` which exist in
`kibana-siem-migrations-rules-<space_id>` but do not exist in
`kibana-siem-migrations-migrations-<space_id>`
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.
Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.
- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Resolves: https://github.com/elastic/security-team/issues/10094
- AgentlessApiResponse type is renamed to AgentlessApiDeploymentResponse
- AgentlessApiDeploymentResponse type matches the Agentless API's
/deployment endpoint response
- Unit tests mocks are adjusted to match the new response.
## Release Notes
Kibana logging's pattern layout, used by default for the console
appender, will now use a new default pattern layout
`[%date][%level][%logger] %message %error`. This will include the error
name and stack trace if these were included in the log entry. To opt out
of this behavior users can omit the `%error` placeholder from their log
pattern config in kibana.yml e.g.:
```
logging:
appenders:
console:
type: console
layout:
type: pattern
pattern: "[%date][%level][%logger] %message"
```
## Summary
Previously, when we pass the error in meta, the information related to
stacktrace and error message was not available in console. This PR
changed the default pattern to also include error information if it is
provided in meta (similar to the way that the logging happens when error
is directly passed to logger.error).
New pattern: (added `%error` at the end)
```
[%date][%level][%logger] %message %error
```
Here you can see the difference:
Logger:
```
server.logger.error(
`Unable to create Synthetics monitor ${monitorWithNamespace[ConfigKey.NAME]}`,
{ error: e }
);
```
#### Before
#### After
### Alternative
We could also change the MetaConversion and include this information,
but we might have additional meta information which I am not sure if it
is OK to be logged by default. Let me know if you prefer changing
MetaConversion instead of adding a new error conversion.
<details>
<summary>Code changes for MetaConversion</summary>
```
function isError(x: any): x is Error {
return x instanceof Error;
}
export const MetaConversion: Conversion = {
pattern: /%meta/g,
convert(record: LogRecord) {
if (!record.meta) {
return '';
}
const { error, ...rest } = record.meta;
const metaString = Object.keys(rest).length !== 0 ? JSON.stringify(rest) : '';
let errorString = '';
if (isError(record.meta?.error)) {
errorString = record.meta?.error.stack || '';
}
return [metaString, errorString].filter(Boolean).join(' ');
},
};
```
</details>
Here is how adjusting meta will look like in this case:
## Summary
This PR fixes the theme bootstrap when the color mode is configured as
`system`.
It uses the `prefers-color-scheme: dark` media match to check if the
preferred color mode is dark or light and adjust the theme accordingly.
Before that PR, the `system` color mode caused kibana to bootstrap with
a light mode (even with a dark OS). Only a subsequent event/check will
update the `__kbnThemeTag__` to the correct theme. But at that point the
SCSS part of elastic-charts is already loaded with the default light
theme tokens (the chart theme is loaded from `src/core/public/index.ts`)
We can probably also find a way to load the chart theme in a subsequent
stage, when the theme is resolved elsewhere, but the general behaviour
is anyway buggy: we are loading the kibana application with a wrong
theme and only after loading more logic we then switch to the actual
theme and update the `__kbnThemeTag__` global.
fix https://github.com/elastic/kibana/issues/210951
To test the before/after:
- in `main` select the `system` color mode, switch to a Dark mode in
your OS, open Discover and apply a breakdown to the histogram: the
legend text is too dark and invisible
- open this pr, select `system` color mode and do the same steps, the
histogram legend will now shows correctly
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This will improve PR #216197.
A new task is created using the task manager every time a global
parameter is created, edited or deleted.
When params is updated/remove/added it will sync task soon, and task
also continue to run every 10 minutes.
### Execution
In the task we check if params have been changed since last run or param
total have changes since last run, in that case we sync private
locations data with package polices
## Testing
Add few browser synthetics monitors in a test private location
Add/Update or delete a param , make sure relevant agent policy is
updated with params. Go to fleet agent policy, view agent policy yaml
and notice params field for a browser monitor in yaml.
---------
Co-authored-by: Shahzad <shahzad31comp@gmail.com>
## Summary
Sorry for the noise, operation!!
This PR includes the `CONTRIBUTOR` author association for the enablement
trigger for the trigger chromium build Github action to resolve the
issue with the action being skipped when it should not be
[see](1511864734). I'd
previously thought `MEMBER` was the way to go, and then `COLLABORATOR`
but on digging into the event Github sent for issue related events for
my user, I see that my author association is `CONTRIBUTOR` [see
here](1771883830),
making this change ensures that only users within the elastic
organisation are able to kick off this action.
In this PR also `jq` is leveraged to compact the json we expect in place
of `sed`.
## Testing
- install the [github local
actions](https://sanjulaganepola.github.io/github-local-actions-docs/)
vscode extension
- copy the response from the linked api response, and save it's contents
as your payload.json file
- start the Github action, we'd see that the action is no logger skipped
and we are able to kick off the build.
<!--
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.
Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.
- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...
-->
## Summary
Fixes an issue where defend insights evaluations always returned no
insights resulting in a score of 0.
The `anonymizedEvents` key was renamed to `anonymizedDocuments` in [this
PR](https://github.com/elastic/kibana/pull/216757). The relevant
datasets have already been updated in langsmith.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
Closes https://github.com/elastic/kibana/issues/221157
## Summary
We run a semantic text migration at startup to add the semantic text
field to documents that were created before 8.17.
Before multilingual KB was introduced:
- We created index assets for KB when the AI Assistant flyout opens.
- Even if the user does not set up the KB, they will have a component
template pointing to the custom inference endpoint.
With the introduction of multilingual KB:
- We moved some of the index creation to when setting up the KB.
- We try to do the semantic_text migration at startup. During this
migration, for users who didn't set up the KB but had the index assets
created at startup, the custom inference endpoint will be unavailable.
- But since the migration uses the inference endpoint from the write
index, we try to access an endpoint that's not available.
This is the reason for this error to be logged.
```
Inference endpoint "obs_ai_assistant_kb_inference" not found or unavailable: resource_not_found_exception
Root causes:
resource_not_found_exception: Inference endpoint not found [obs_ai_assistant_kb_inference]
```
There is no customer impact from this, just that the error that gets
logged is creating a lot of noise.
## Solution
This PR checks whether there are documents without semantic_text before
starting the migration.
And also reduced the log level to warn because we hit the `/status`
endpoint once when a user opens the AI Assistant.
### Checklist
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Summary
This is a followup to https://github.com/elastic/kibana/pull/220675
where we added a new `xpack.alerting.rules.run.ruleTypeOverrides` inside
the `serverless.security.yml`.
After some investigation we realized that it overrides the same config
declared in the root `serverless.yml` file. Since the root file contains
the overrides for the security specific configs, we decided to move
those into the `serverless.security.yml`.
More details in [internal
discussion](https://elastic.slack.com/archives/C5TQ33ND8/p1747748907000299).
## Summary
Change the AI Connector description to `Send requests to AI providers
such as Amazon Bedrock, OpenAI and more.`
### Checklist
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
