## Summary
1. Show all packages owned by a specific team
```
node scripts/dependency_ownership -o <owner>
```
2. Identify owners of a specific dependency
```
node scripts/dependency_ownership -d <dependency>
```
3. List dependencies without an owner
```
node scripts/dependency_ownership --missing-owner
```
4. Generate a full dependency ownership report
```
node scripts/dependency_ownership
```
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
__Closes: https://github.com/elastic/kibana/issues/196767__
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Creates the Streams app plugin, which renders UI for managing streams
(see https://github.com/elastic/kibana/pull/198713).
Additional changes in this PR:
- The menus were updated to conditionally add a link to the Streams app.
The Streams plugin itself returns a status$ observable which signals if
Streams have been enabled. This value is used to conditionally render
the link in the various flavors of menus.
- There's a small change in the ES types to allow for ordered params in
ES|QL (vs named params)
- `@kbn/server-route-repository` was updated to be able to override
`access` (instead of only inferring it from the endpoint name).
Additionally, we now allow all route options by default.
- `@kbn/typed-react-router-config` now also exports a `useBreadcrumbs`.
This was copied over from the APM implementation.
- the signature of the `esql` method in
`ObservabilityElasticsearchClient` was updated to separate processing
options from options that are sent over to the _query endpoint.
---------
Co-authored-by: Chris Cowan <chris@elastic.co>
Co-authored-by: Joe Reuter <johannes.reuter@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
[dependency-cruiser](https://github.com/sverweij/dependency-cruiser/tree/main)
is used for building dependency graph.
### Show all dependencies for a specific package/plugin or directory
#### Run for all plugins
```bash
bash scripts/dependency_usage.sh -p x-pack/plugins -o ./tmp/deps-result-all.json
```
#### Run for single plugin
```bash
bash scripts/dependency_usage.sh -p x-pack/plugins/security_solution -o ./tmp/deps-result-single.json
```
#### Run for multiple plugins
```bash
bash scripts/dependency_usage.sh -p x-pack/plugins/security_solution x-pack/plugins/security -o ./tmp/deps-result-multiple.json
```
#### Run for `x-pack/packages`
```bash
bash scripts/dependency_usage.sh -p x-pack/packages -o ./tmp/deps-packages-1.json
```
#### Run for `packages`
```bash
bash scripts/dependency_usage.sh -p packages -o ./tmp/deps-packages-2.json
```
#### Benchmark
| Analysis | Real Time | User Time | Sys Time |
|-----------------------|-------------|-------------|------------|
| All plugins | 7m 21.126s | 7m 53.099s | 20.581s |
| Single plugin | 31.360s | 45.352s | 2.208s |
| Multiple plugins | 36.403s | 50.563s | 2.814s |
| x-pack/packages | 6.638s | 12.646s | 0.654s |
| packages | 25.744s | 39.073s | 2.191s |
#### Show all packages/plugins within a directory that use a specific
dependency
```sh
bash scripts/dependency_usage.sh -d rxjs -p x-pack/plugins/security_solution
```
---
#### Show all packages/plugins within a directory grouped by code owner
```sh
bash scripts/dependency_usage.sh -d rxjs -p x-pack/plugins -g owner
```
---
#### Group by code owner with adjustable collapse depth for fine-grained
grouping
**Fine-grained grouping**:
```sh
bash scripts/dependency_usage.sh -p x-pack/plugins/security_solution -g owner --collapse-depth 4
```
**Collapsed grouping**: groups the results under a higher-level owner
(e.g., `security_solution` as a single group).
```bash
bash scripts/dependency_usage.sh -p x-pack/plugins/security_solution -g owner --collapse-depth 1
```
---
#### Show all dependencies matching a pattern (e.g., `react-*`) within a
package
```bash
bash scripts/dependency_usage.sh -p x-pack/plugins/security_solution -d 'react-*' -o ./tmp/result.json
```
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
__Related: https://github.com/elastic/kibana/issues/196767__
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
This PR makes the `vis-type-timeseries-plugin` a shared component fixing
the dependency error for infra plugin.
I removed the infra folder from the exceptions list because it was
preventing the linter from acknowledging the fix.
### Summary
This PR relocates some plugins and packages that are incorrectly
categorised, aiming at reducing inter-solution dependencies. It also
fixes some incorrect import statements that introduce unnecessary
dependencies with other components.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Maryam Saeidi <maryam.saeidi@elastic.co>
## Summary
This PR aims to fix Flaky tests related to agentless detected by
https://github.com/elastic/kibana/issues/189038 and
https://github.com/elastic/kibana/issues/192126 by adding proper
handling of the `waitFor` methods.
It was also detected with
https://github.com/elastic/security-team/issues/10979 that some other
methods were not proper handled by `waitFor`, leading to the assertions
inside those unhandled `waitFor` being skipped by Jest.
This PR also introduces ESLint to enforce proper handling of waitFor
methods in tests files for Fleet and Cloud Security plugins.
Additional note: These changes should also unblock the failing tests on
the [React18 use waitFor with assertion callbacks in place of
waitForNextUpdate](https://github.com/elastic/kibana/pull/195087) PR
**Fleet changes**
- ESLint rule added to enforce handling `waitFor` on React Testing
Library.
- `useSetupTechnology` hook tests reviewed and updated to handle the
waitFor. Fixed issue identified when reviewing the tests.
- step_define_package_policy.test.tsx: Added package policy vars to the
mock to proper handle the use cases
- step_select_hosts.test.tsx: Handled waitFor, identified outdated test
- step_edit_hosts.test.tsx: Handled waitFor, identified outdated test
With the introduction of the ESLint rule other tests were triggering
ESLint errors, I attempted to fix them while retaining the same
intention, let me know if more changes are needed.
**Cloud Security changes**
- ESLint rule added to enforce handling `waitFor` on React Testing
Library.
- Updated cloud security posture version to include agentless global
tags on End to End tests
**@elastic/kibana-operations changes**
- Added
[eslint-plugin-testing-library](https://testing-library.com/docs/ecosystem-eslint-plugin-testing-library/)
an ESLint plugin for Testing Library that helps users to follow best
practices and anticipate common mistakes when writing tests.
- The adoption and enablement of the rules are opt-in.
## Summary
Follow-up of https://github.com/elastic/kibana/pull/195367
As part of the Sustainable Kibana Architecture initiative, this PR
leverages the mechanisms and concepts introduced in
https://github.com/elastic/kibana/pull/194810, updating plugins that
were considered to be solution-specific in Luke's
[PoC](https://github.com/elastic/kibana/pull/179710).
This might trigger linting rule violations in CI, and help uncover
conflicts related to forbidden dependencies.
As soon as they are resolved, we can proceed to classify solutions'
plugins.
# Summary
Adds a new API deprecations feature inside core.
This feature enabled plugin developers to mark their versioned and
unversioned public routes as deprecated.
These deprecations will be surfaced to the users through UA to help them
understand the deprecation and address it before upgrading. This PR also
surfaces these deprecations to UA.
Closes https://github.com/elastic/kibana/issues/117241
1. Core service to flag deprecated routes
2. UA code to surface and resolve deprecated routes
## Flagging a deprecated Route
### The route deprecation option
We have three types of route deprecations:
- `type: bump`: A version bump deprecation means the API has a new
version and the current version will be removed in the future in favor
of the newer version.
- `type: remove`: This API will be completely removed. You will no
longer be able to use it in the future.
- `type: migrate`: This API will be migrated to a different API and will
be removed in the future in favor of the other API.
All route deprecations expect a documentation link to help users
navigate. We might add a generic documentation link and drop this
requirement in the future but for now this is required.
### Deprecated Route Example
Full examples can be found in the `routing_example` example plugin
located in this directory:
`examples/routing_example/server/routes/deprecated_routes`
```ts
router[versioned?].get(
{
path: '/',
options: {
deprecated: {
documentationUrl: 'https://google.com',
severity: 'warning',
reason: {
type: 'bump',
newApiVersion: '2024-10-13',
},
},
},
},
async (context, req, res) => {
...
```
## Surfaced API deprecations in UA
The list of deprecated APIs will be listed inside Kibana deprecations
along with the already supported config deprecations.
<img width="1728" alt="image"
src="https://github.com/user-attachments/assets/5bece704-b80b-4397-8ba2-6235f8995e4a">
Users can click on the list item to learn more about each deprecation
and mark it as resolved
<img width="1476" alt="image"
src="https://github.com/user-attachments/assets/91c9207b-b246-482d-a5e4-21d0c61582a8">
### Marking as resolved
Users can click on mark as resolved button in the UA to hide the
deprecation from the Kiban deprecations list.
We keep track on when this button was clicked and how many times the API
has been called. If the API is called again the deprecation will
re-appear inside the list. We might add a feature in the future to
permenantly supress the API deprecation from showing in the list through
a configuration (https://github.com/elastic/kibana/issues/196089)
If the API has been marked as resolved before we show this in the flyout
message:
> The API GET /api/deprecations/ has been called 25 times. The last time
the API was called was on Monday, October 14, 2024 1:08 PM +03:00.
> The api has been called 2 times since the last time it was marked as
resolved on Monday, October 14, 2024 1:08 PM +03:00
Once marked as resolved the flyout exists and we show this to the user
until they refresh the page
<img width="1453" alt="image"
src="https://github.com/user-attachments/assets/8bb5bc8b-d1a3-478f-9489-23cfa7db6350">
## Telemetry:
We keep track of 2 new things for telemetry purposes:
1. The number of times the deprecated API has been called
2. The number of times the deprecated API has been resolved (how many
times the mark as resolved button in UA was clicked)
## Code review
- [x] Core team is expected to review the whole PR
- [ ] Docs team to review the copy and update the UA displayed texts
(title, description, and manual steps)
- [x] kibana-management team is expected to review the UA code changes
and UI
- [ ] A few teams are only required to approve this PR and update their
`deprecated: true` route param to the new deprecationInfo object we now
expect. There is an issue tracker to address those in separate PRs later
on: https://github.com/elastic/kibana/issues/196095
## Testing
Run kibana locally with the test example plugin that has deprecated
routes
```
yarn start --plugin-path=examples/routing_example --plugin-path=examples/developer_examples
```
The following comprehensive deprecated routes examples are registered
inside the folder:
`examples/routing_example/server/routes/deprecated_routes`
Run them in the console to trigger the deprecation condition so they
show up in the UA:
```
# Versioned routes: Version 1 is deprecated
GET kbn:/api/routing_example/d/versioned?apiVersion=1
GET kbn:/api/routing_example/d/versioned?apiVersion=2
# Non-versioned routes
GET kbn:/api/routing_example/d/removed_route
POST kbn:/api/routing_example/d/migrated_route
{}
```
1. You can also mark as deprecated in the UA to remove the deprecation
from the list.
2. Check the telemetry response to see the reported data about the
deprecated route.
3. Calling version 2 of the API does not do anything since it is not
deprecated unlike version `1` (`GET
kbn:/api/routing_example/d/versioned?apiVersion=2`)
4. Internally you can see the deprecations counters from the dev console
by running the following:
```
GET .kibana_usage_counters/_search
{
"query": {
"bool": {
"should": [
{"match": { "usage-counter.counterType": "deprecated_api_call:total"}},
{"match": { "usage-counter.counterType": "deprecated_api_call:resolved"}},
{"match": { "usage-counter.counterType": "deprecated_api_call:marked_as_resolved"}}
]
}
}
}
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: florent-leborgne <florent.leborgne@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
I noticed that our `no-restricted-imports` rules were not working on
some parts of the codebase. Turns our the rule was overriden by mistake.
This PR fixes the rules and places that were not following them:
- lodash set for safety
- react-use for a bit smaller bundles
- router for context annoncement (`useExecutionContext`) and hopefully
easier upgrade to newer version
This is a re-submission of https://github.com/elastic/kibana/pull/191899, which was reverted due to
a storybook build problem. This introduces a "Logs Overview" component for use in solution UIs
behind a feature flag.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Kerry Gallagher <471693+Kerry350@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This introduces a "Logs Overview" component for use in solution UIs
behind a feature flag.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Kerry Gallagher <471693+Kerry350@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Follow-up to #190690
Most of API integration tests does not match the path pattern set in the
original PR (thanks @pheyos for catching it) and where not updated.
This PR updates `.eslintrc.js` with explicit patterns to lint
api_integration tests. Hopefully it is final change, but I rely on code
owners to double check it.
Most of the changes are trivial adjustments:
- duplicated before/after hooks `mocha/no-sibling-hooks`
- duplicated test titles `mocha/no-identical-title`
- async function in describe() `mocha/no-async-describe`
---------
Co-authored-by: Ash <1849116+ashokaditya@users.noreply.github.com>
Adds a `@kbn/observability-utils` package.
```md
# @kbn/observability-utils
This package contains utilities for Observability plugins. It's a separate package
to get out of dependency hell. You can put anything in here that is stateless and
has no dependency on other plugins (either directly or via other packages).
The utility functions should be used via direct imports to minimize impact on
bundle size and limit the risk on importing browser code to the server and vice versa.
```
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
This PR enforces ESLint rules in FTR tests, in particular:
- `no-floating-promises` rule to catch unawaited Promises in
tests/services/page objects
_Why is it important?_
- Keep correct test execution order: cleanup code may run before the
async operation is completed, leading to unexpected behavior in
subsequent tests
- Accurate test results: If a test completes before an async operation
(e.g., API request) has finished, Mocha might report the test as passed
or failed based on incomplete context.
```
198:11 error Promises must be awaited, end with a call to .catch, end with a call to .then
with a rejection handler or be explicitly marked as ignored with the `void` operator
@typescript-eslint/no-floating-promises
```
<img width="716" alt="Screenshot 2024-08-20 at 14 04 43"
src="https://github.com/user-attachments/assets/9afffe4c-4b51-4790-964c-c44a76baed1e">
- recommended rules from
[eslint-mocha-plugin](https://www.npmjs.com/package/eslint-plugin-mocha)
including:
-
[no-async-describe](https://github.com/lo1tuma/eslint-plugin-mocha/blob/main/docs/rules/no-async-describe.md)
-
[no-identical-title.md](https://github.com/lo1tuma/eslint-plugin-mocha/blob/main/docs/rules/no-identical-title.md)
-
[no-sibling-hooks.md](https://github.com/lo1tuma/eslint-plugin-mocha/blob/main/docs/rules/no-sibling-hooks.md)
and others
Note for reviewers: some tests were skipped due to failures after
missing `await` was added. Most likely is a "false positive" case when
test is finished before async operation is actually completed. Please
work on fixing and re-enabling it
---------
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
### This PR introduces a new type of API integration tests in FTR:
deployment-agnostic
#### Test suite is considered deployment-agnostic when it fulfils the
following criteria:
**Functionality**: It tests Kibana APIs that are **logically identical
in both stateful and serverless environments** for the same SAML roles.
**Design**: The test design is **clean and does not require additional
logic** to execute in either stateful or serverless environments.
### How It Works
Most existing stateful tests use basic authentication for API testing.
In contrast, serverless tests use SAML authentication with
project-specific role mapping.
Since stateful deployments also support SAML, deployment-agnostic tests
**configure Elasticsearch and Kibana with SAML authentication in both
cases**. For roles, stateful deployments define 'viewer', 'editor', and
'admin' roles with serverless-alike privileges.
New `samlAuth` service has `AuthProvider` interface with 2 different
implementations: depending on environment context (serverless or
stateful) appropriate implementation is used. But it remains on service
level and hidden in test suite.
test example
```
export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
const samlAuth = getService('samlAuth');
const supertestWithoutAuth = getService('supertestWithoutAuth');
let roleAuthc: RoleCredentials;
let internalHeaders: InternalRequestHeader;
describe('GET /api/console/api_server', () => {
before(async () => {
roleAuthc = await samlAuth.createM2mApiKeyWithRoleScope('admin');
internalHeaders = samlAuth.getInternalRequestHeader();
});
after(async () => {
await samlAuth.invalidateM2mApiKeyWithRoleScope(roleAuthc);
});
it('returns autocomplete definitions', async () => {
const { body } = await supertestWithoutAuth
.get('/api/console/api_server')
.set(roleAuthc.apiKeyHeader)
.set(internalHeaders)
.set('kbn-xsrf', 'true')
.expect(200);
expect(body.es).to.be.ok();
const {
es: { name, globals, endpoints },
} = body;
expect(name).to.be.ok();
expect(Object.keys(globals).length).to.be.above(0);
expect(Object.keys(endpoints).length).to.be.above(0);
});
});
}
```
Please read
[readme](966822ac87/x-pack/test/api_integration/deployment_agnostic/README.md)
for more details and step-by-step guide. It should help migrating
existing serverless tests to deployment-agnostic, assuming requirements
are met.
### Examples
Deployment-agnostic tests:
```
x-pack/test/api_integration/deployment_agnostic/apis/console/spec_definitions.ts
x-pack/test/api_integration/deployment_agnostic/apis/core/compression.ts
x-pack/test/api_integration/deployment_agnostic/apis/painless_lab/painless_lab.ts
```
Configs to run it:
```
node scripts/functional_tests --config x-pack/test/api_integration/deployment_agnostic/oblt.serverless.config.ts
node scripts/functional_tests --config x-pack/test/api_integration/deployment_agnostic/search.serverless.config.ts
node scripts/functional_tests --config x-pack/test/api_integration/deployment_agnostic/security.serverless.config.ts
node scripts/functional_tests --config x-pack/test/api_integration/deployment_agnostic/stateful.config.ts
```
PR is a compact version of #188737 with reduced changes in existing
serverless tests.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: elena-shostak <165678770+elena-shostak@users.noreply.github.com>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>
## Summary
This ticket is the initial implementation for the UI side for the
AI-driven custom integration creation.
This PR only contains the implementation of the UI, due to the tight
timing it will not include tests, everything will be tested manually for
8.15 FF. We'll implement the tests later.
#### Enable Feature
The new integration assistant plugin is disabled by default, to enable
it:
```
xpack.integration_assistant.enabled: true
```
#### Complete tasks
- [x] New integration button on the /integrations page
- [x] New integration "landing" page with buttons to upload zip and
assistant
- [x] Upload zip page to install integration
- [x] Integration assistant:
- [x] Connector selection step
- [x] Integration details step
- [x] Data stream step
- [x] Review and install
#### Follow-ups (will be implemented in separate PRs)
- [ ] Add RBAC
- [ ] Add telemetry
- [ ] Documentation
- [ ] Add license/productType controls
- [ ] Add links to the create integration page
- [ ] Improve package name retrieval:
https://github.com/elastic/kibana/issues/185932
- [ ] Add time estimation on the generation stage
- [ ] Add support for multi-valuated "input type"
- [ ] Enable Langsmith tracing using AI assistant settings
#### Demo
b04c21c6-09cf-49bb-be8f-bf4b9d3feb8e
## Files by Code Owner
### elastic/docs
* packages/kbn-doc-links/src/get_doc_links.ts
* packages/kbn-doc-links/src/types.ts
### elastic/fleet
* x-pack/plugins/fleet/kibana.jsonc
*
x-pack/plugins/fleet/public/applications/integrations/hooks/use_breadcrumbs.tsx
*
x-pack/plugins/fleet/public/applications/integrations/layouts/default.tsx
*
x-pack/plugins/fleet/public/applications/integrations/sections/epm/index.tsx
*
x-pack/plugins/fleet/public/applications/integrations/sections/epm/screens/create/index.tsx
* x-pack/plugins/fleet/public/components/header.tsx
* x-pack/plugins/fleet/public/constants/page_paths.ts
* x-pack/plugins/fleet/public/plugin.ts
* x-pack/plugins/fleet/tsconfig.json
### elastic/kibana-core
* x-pack/plugins/fleet/kibana.jsonc
* x-pack/plugins/integration_assistant/kibana.jsonc
### elastic/kibana-operations
* packages/kbn-optimizer/limits.yml
### elastic/security-solution
* x-pack/plugins/integration_assistant/**/*
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This is a PR to add a new backend plugin (frontend will be done in
separate [PR](https://github.com/elastic/kibana/pull/184546)).
The purpose of the plugin is to provide a set of API routes that is used
to perform a variety of GenAI workflows to generate new integrations
based on provided inputs.
It reuses the existing GenAI connectors for its LLM communication, and
provides a set of API's to create ECS mapping, Categorization, Related
Fields and an API to generate the actual integration package zip, which
is forwarded to the UI component.
### Planned follow-up changes:
As the PR is getting way too large, some planned changes would be added
in much smaller follow-ups. This includes mostly more improved try/catch
for certain routes, adding debug/error log entries where relevant,
especially for the API endpoints themself, some more unit and end2end
tests.
- OpenAPI spec for the API will be handled in a separate PR
- All the missing unit tests will be added as a followup PR
### Testing
The `integration_assistant` plugin will be disabled by default while
it's being implemented so we can iterate and merge partial PRs without
interfering with the releases. This config will work as our feature
flag:
6aefd4ff7b/x-pack/plugins/integration_assistant/server/config.ts (L11-L13)
To test it add this to your _kibana.dev.yml_:
```
xpack.integration_assistant.enabled: true
```
### Checklist
Delete any items that are not applicable to this PR.
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: Patryk Kopycinski <contact@patrykkopycinski.com>
Co-authored-by: Sergi Massaneda <sergi.massaneda@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Bharat Pasupula <saibharatchandra.pasupula@elastic.co>
Co-authored-by: Bharat Pasupula <123897612+bhapas@users.noreply.github.com>
Use one react-hooks/exhaustive-deps across our Obs plugins, for
consistency reasons.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Carlos Crespo <crespocarlos@users.noreply.github.com>
## Summary
Upgrading intl packages from v2 to v6
### Packages upgrade:
- [x] Add @formatJS packages
- [x] `react-intl` Upgraded
- [x] `intl-messageformat` Upgraded
- [x] `intl-format-cache` removed
- [x] `intl-relativeformat` removed
- [x] `intl-messageformat-parser` removed
### Todo list:
- [x] Refactor HTML tags
- [x] Refactor Upgrade tags
- [x] Refactor `kbn-i18n`
- [x] Refactor `kbn-i18n-react`
- [x] Refactor `FormattedRelative` to `FormattedRelativeTime`
- [x] Refactor polyfills
- [x] Refactor IntlShape types
- [x] Rewrite Providers
- [x] Rewrite tests using i18n
- [x] Removed current pseudolocale implementation (tracker:
https://github.com/elastic/kibana/issues/180244)
- [x] Fix jest tests using rendered `Provider`
- [x] Remove no longer valid i18n packages documentation (tracker:
https://github.com/elastic/kibana/issues/180259)
Closes https://github.com/elastic/kibana/issues/178968
Closes https://github.com/elastic/kibana/issues/38642
## Notes to code reviewers
For team other than the core team, please review your plugins code
changes by filtering files by codeowners.
### Test Snapshot updates
Most of the changes are refactors of renamed functions and changed ICU
syntax.
The main updates are snapshot changes where `FormattedMessage` is now
memoized so snapshots capturing the html tree needed to be updated to
use `<Memo(MemoizedFormattedMessage)` instead of `<FormattedMessage`
### ICU now supports HTML tags:
before:
```
<FormattedMessage
defaultMessage="To buy a shoe, { link } and { cta }"
values={{
link: (
<a class="external_link" target="_blank" href="https://www.shoe.com/">
visit our website
</a>
),
cta: <strong class="important">eat a shoe</strong>,
}}
/>
```
after:
```
<FormattedMessage
defaultMessage="To buy a shoe, <a>visit our website</a> and <cta>eat a shoe</cta>"
values={{
a: msg => (
<a class="external_link" target="_blank" href="https://www.shoe.com/">
{msg}
</a>
),
cta: msg => <strong class="important">{msg}</strong>,
}}
/>
```
### Escape character to prevent ICU parsing changed from double slashes
to single quotes:
before: `\\{escaped\\}`
after: `'{escaped}'`
### No need for Intl Shape
the new packages under formatJS are written in typescript and come with
types support out of the box so no need to set types when using i18n.
Renamed `InjectedIntlProps` with `WrappedComponentProps`.
Removed `prop-types` and `intlShape` in favor of `IntlShape`.
### FormattedRelative has been renamed to FormattedRelativeTime and its
API has changed significantly. See
[FormattedRelativeTime](https://formatjs.io/docs/react-intl/upgrade-guide-3x#formattedrelativetime)
for more details.
### All tags specified must have corresponding values and will throw
error if it's missing
All tags are now parsed and expected to be formatted properly (all
opened tags must be closed).
To skip this check you can use the `ignoreTag: true` property
```
i18n.translate('xpack.apm.agentConfig.captureJmxMetrics.description', {
defaultMessage: 'This is not an HTML tag <JMX object name pattern>' +
ignoreTag: true,
}),
```
**When do I use ignore tags?**
If your message has HTML tags, it is preferred not to ignore the Tag to
have some string verification that the html tags you are adding are
properly formatted and closed.
If it the text between brackets is not an HTML tag and it is just a
fomat preference then using `ignoreTag` makes sense.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
## Summary
Close https://github.com/elastic/kibana/issues/182390
### To test
You can test all aspects of the script by first making some changes to
`packages/kbn-esql-validation-autocomplete/src/validation/validation.test.ts`
- delete several tests from an existing function describe block ("block
1"—except don't choose "date_diff" since it has a bunch of custom tests)
- delete another function describe block completely ("block 2")
- change the expected result of several of the tests in a third function
describe block ("block 3")
Then, run `yarn maketests` from within
`packages/kbn-esql-validation-autocomplete`
**Expected result**
- Block 1 should have the deleted tests restored
- Block 2 should be restored entirely (though it may be moved in the
tests file)
- Block 3 should be untouched
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Adds the same Eslint rules as the rest of security_solution to the
`cell-actions` package, adapting the code.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Enables telemetry warning eslint rule.
Fixes a bunch of missing telemetry strings.
### Checklist
Delete any items that are not applicable to this PR.
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
Fixes https://github.com/elastic/kibana/issues/176420
## 🍒 Summary
This PR copies the SLO code that was inside the Observability app into
its own app under `observability-solution/slo` folder.
4f6b8dfb-9612-4d30-ad50-4ee5c55a9c32
## ✔️ Acceptance criteria
- URL of new app: `app/slos`
- Design and functionality are not changed.
- Git history has been retained for all files in
`x-pack/plugins/observability_solution/slo`.
- SLO should appear on server less
- SLO code inside `observability_solution/observability` code has been
removed. A new clean up round might be needed though for possible
leftovers.
- Burn rate rule is registered within the new slo app
- SLO embeddables are moved inside the new slo app
- overview
- alerts embeddable
- error budget burn down
- Alerts table configuration registration for slo details page and
alerts table embeddable is still done in the observability app. Response
Ops team is working on removing the need to register the alert table
anyway
- Slo app is wrapped into `ApplicationUsageTrackingProvider` which will
send slo `Application usage` information tracked by the `slo` appId
- Redirect old `app/observability/slos` route to `app/slos`
- Rename old `xpack.observability.slo` keys to `xpack.slo` in the
translation files
## 🌮 How to test
Design and functionality didn't change, so simply navigate to existing
slo pages and try to break it
- Slo list page
- group by
- unified search
- toggle buttons
- actions
- Slo creation
- try group by as well
- Slo detail page
- Actions on top
- navigate to overview and alerts tabs
- Create SLO flyout in Logs Explorer
- Create burn rate rules and verify they appear on rules page
- Verify SLO alerts appear on Alerts page and slo details page
- Embeddables
- Through the dashboard app
- Using the attach to dashboard action on the slo card item on slo list
page and the error budget burn down chart on the slo detail page
- SLOs only for platinum users
- Permissions
- Spaces
## TODO
- [x] Move slo stuff from observability folder to new slo plugin
- [x] Remove old slo stuff from observability folder
- [x] Update references
- [x] Fix typescript and eslint errors
- [x] Paths
- [x] Locators
- [x] Burn rate rule registration
- [x] Embeddable Alerts table configuration registration
- [x] Embeddables
- [x] Translations
- [x] Verify plugin.ts files contain all registration logic
- [x] public
- [x] server
- [x] Final cleanup for observability folder
- [x] Run tests
- [x] Application Usage (Telemetry)
- [x] Permissions
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: shahzad31 <shahzad31comp@gmail.com>
Co-authored-by: Coen Warmer <coen.warmer@gmail.com>
## Summary
issue https://github.com/elastic/security-team/issues/7382
### Data Stream Adapter
This PR introduces the `@kbn/data-stream-adapter` package, which is a
utility library to facilitate Data Stream creation and maintenance in
Kibana, it was inspired by the data stream implementation in the Alerts
plugin.
The library has two exports:
- `DataStreamSpacesAdapter`: to manage space data streams. It uses the
`name-of-the-data-stream-<spaceId>` naming pattern.
- `DataStreamAdapter`: to manage single (not space-aware) data streams.
Usage examples in the package
[README](450be0369d/packages/kbn-data-stream-adapter/README.md)
### Data Quality Dashboard
The `DataStreamSpacesAdapter` has been integrated into the data quality
dashboard to store all the quality checks users perform. The information
stored is the metadata (also used for telemetry) and the actual data
rendered in the tables.
FieldMap definition
[here](450be0369d/x-pack/plugins/ecs_data_quality_dashboard/server/lib/data_stream/results_field_map.ts)
### Demo
311a0bf5-004b-46d7-8140-52a233361c91
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Philippe Oberti <philippe.oberti@elastic.co>
Co-authored-by: Garrett Spong <spong@users.noreply.github.com>
Co-authored-by: Efe Gürkan YALAMAN <efeguerkan.yalaman@elastic.co>
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
Co-authored-by: Sander Philipse <94373878+sphilipse@users.noreply.github.com>
Co-authored-by: JD Kurma <JDKurma@gmail.com>
Co-authored-by: Jan Monschke <jan.monschke@elastic.co>
Co-authored-by: Patryk Kopyciński <contact@patrykkopycinski.com>
Co-authored-by: Khristinin Nikita <nikita.khristinin@elastic.co>
Co-authored-by: Marco Liberati <dej611@users.noreply.github.com>
Co-authored-by: Julia Rechkunova <julia.rechkunova@elastic.co>
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
Co-authored-by: Davis McPhee <davis.mcphee@elastic.co>
Co-authored-by: Eyo O. Eyo <7893459+eokoneyo@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Marta Bondyra <4283304+mbondyra@users.noreply.github.com>
Co-authored-by: Søren Louv-Jansen <soren.louv@elastic.co>
Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
Co-authored-by: Candace Park <56409205+parkiino@users.noreply.github.com>
## Summary
Refactoring general ui service to a kbn package.
Resolves an [Appex QA](https://github.com/elastic/appex-qa-team) issue.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## [Security Solution] [Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts
This PR implements _Retrieval Augmented Generation_ (RAG) for Alerts in the Security Solution. This feature enables users to ask the assistant questions about the latest and riskiest open alerts in their environment using natural language, for example:
- _How many alerts are currently open?_
- _Which alerts should I look at first?_
- _Did we have any alerts with suspicious activity on Windows machines?_
### More context
Previously, the assistant relied solely on the knowledge of the configured LLM and _singular_ alerts or events passed _by the client_ to the LLM as prompt context. This new feature:
- Enables _multiple_ alerts to be passed by the _server_ as context to the LLM, via [LangChain tools](https://github.com/elastic/kibana/pull/167097)
- Applies the user's [anonymization](https://github.com/elastic/kibana/pull/159857) settings to those alerts
- Only fields allowed by the user will be sent as context to the LLM
- Users may enable or disable anonymization for specific fields (via settings)
- Click the conversation's `Show anonymized` toggle to see the anonymized values sent to / received from the LLM:
### Settings
This feature is enabled and configured via the `Knowledge Base` > `Alerts` settings in the screenshot below:
- The `Alerts` toggle enables or disables the feature
- The slider has a range of `10` - `100` alerts (default: `20`)
When the setting above is enabled, up to `n` alerts (as determined by the slider) that meet the following criteria will be returned:
- the `kibana.alert.workflow_status` must be `open`
- the alert must have been generated in the last `24 hours`
- the alert must NOT be a `kibana.alert.building_block_type` alert
- the `n` alerts are ordered by `kibana.alert.risk_score`, to prioritize the riskiest alerts
### Feature flag
To use this feature:
1) Add the `assistantRagOnAlerts` feature flag to the `xpack.securitySolution.enableExperimental` setting in `config/kibana.yml` (or `config/kibana.dev.yml` in local development environments), per the example below:
```
xpack.securitySolution.enableExperimental: ['assistantRagOnAlerts']
```
2) Enable the `Alerts` toggle in the Assistant's `Knowledge Base` settings, per the screenshot below:
## How it works
- When the `Alerts` settings toggle is enabled, http `POST` requests to the `/internal/elastic_assistant/actions/connector/{id}/_execute` route include the following new (optional) parameters:
- `alertsIndexPattern`, the alerts index for the current Kibana Space, e.g. `.alerts-security.alerts-default`
- `allow`, the user's `Allowed` fields in the `Anonymization` settings, e.g. `["@timestamp", "cloud.availability_zone", "file.name", "user.name", ...]`
- `allowReplacement`, the user's `Anonymized` fields in the `Anonymization` settings, e.g. `["cloud.availability_zone", "host.name", "user.name", ...]`
- `replacements`, a `Record<string, string>` of replacements (generated on the server) that starts empty for a new conversation, and accumulates anonymized values until the conversation is cleared, e.g.
```json
"replacements": {
"e4f935c0-5a80-47b2-ac7f-816610790364": "Host-itk8qh4tjm",
"cf61f946-d643-4b15-899f-6ffe3fd36097": "rpwmjvuuia",
"7f80b092-fb1a-48a2-a634-3abc61b32157": "6astve9g6s",
"f979c0d5-db1b-4506-b425-500821d00813": "Host-odqbow6tmc",
// ...
},
```
- `size`, the numeric value set by the slider in the user's `Knowledge Base > Alerts` setting, e.g. `20`
- The `postActionsConnectorExecuteRoute` function in `x-pack/plugins/elastic_assistant/server/routes/post_actions_connector_execute.ts` was updated to accept the new optional parameters, and to return an updated `replacements` with every response. (Every new request that is processed on the server may add additional anonymized values to the `replacements` returned in the response.)
- The `callAgentExecutor` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/execute_custom_llm_chain/index.ts` previously used a hard-coded array of LangChain tools that had just one entry, for the `ESQLKnowledgeBaseTool` tool. That hard-coded array was replaced in this PR with a call to the (new) `getApplicableTools` function:
```typescript
const tools: Tool[] = getApplicableTools({
allow,
allowReplacement,
alertsIndexPattern,
assistantLangChain,
chain,
esClient,
modelExists,
onNewReplacements,
replacements,
request,
size,
});
```
- The `getApplicableTools` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/index.ts` examines the parameters in the `KibanaRequest` and only returns a filtered set of LangChain tools. If the request doesn't contain all the parameters required by a tool, it will NOT be returned by `getApplicableTools`. For example, if the required anonymization parameters are not included in the request, the `open-alerts` tool will not be returned.
- The new `alert-counts` LangChain tool returned by the `getAlertCountsTool` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/alert_counts/get_alert_counts_tool.ts` provides the LLM the results of an aggregation on the last `24` hours of alerts (in the current Kibana Space), grouped by `kibana.alert.severity`. See the `getAlertsCountQuery` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/alert_counts/get_alert_counts_query.ts` for details
- The new `open-alerts` LangChain tool returned by the `getOpenAlertsTool` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/open_alerts/get_open_alerts_tool.ts` provides the LLM up to `size` non-building-block alerts generated in the last `24` hours (in the current Kibana Space) with an `open` workflow status, ordered by `kibana.alert.risk_score` to prioritize the riskiest alerts. See the `getOpenAlertsQuery` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/open_alerts/get_open_alerts_query.ts` for details.
- On the client, a conversation continues to accumulate additional `replacements` (and send them in subsequent requests) until the conversation is cleared
- Anonymization functions that were only invoked by the browser were moved from the (browser) `kbn-elastic-assistant` package in `x-pack/packages/kbn-elastic-assistant/` to a new common package: `x-pack/packages/kbn-elastic-assistant-common`
- The new `kbn-elastic-assistant-common` package is also consumed by the `elastic_assistant` (server) plugin: `x-pack/plugins/elastic_assistant`
## Summary
This PR adds a bunch of plugins to help manage AI Assistant Management
settings.
It offers a 'selection' plugin inside Stack Management where a user can
select which AI Assistant she wants to manage.
The Security team can hook into this one, so settings for both AI
Assistants can be accessed from inside one place inside Stack
Management.
This PR also adds the plugin to manage settings for the AI Assistant for
Observability, including Knowledge Base management. This plugin is
available both in Stack Management (stateful) and Project Settings
(serverless).
## What it looks like
51392ec5-05c9-4947-9bf2-810d8d0b7525
## Detailed
1. **Adds a Stack Management plugin**
(`/src/plugins/ai_assistant_management/selection`). Its primary function
is to render a selection screen to help users navigate to the settings
plugin for the AI Assistant for a specific solution. This plugin is
displayed in Stack Management, which is only available in stateful
versions of Kibana.
2. **Adds a AI Assistant for Observability Settings plugin**
(`/src/plugins/ai_assistant_management/observability`). This plugin
allows management of specific Observability AI Assistant settings. It is
available in stateful versions of Kibana (via the aforementioned Stack
Management plugin) or in serverless versions via Project Management.
3. **Knowledge Base management for Observability AI Assistant**: The AI
Assistant for Observability Settings plugin has a Knowledge Base tab,
which allows users to add / read / update / delete and bulk import
entries into the Knowledge Base of the Observability AI Assistant.
4. **Moving of KB endpoints in Observability AI Assistant plugin**: KB
endpoints and functions were located in the same folder. As this PR adds
new endpoints for the KB for CRUD operations, it also moves the existing
ones from the function folder into a dedicated one so there's a clearer
distinction between kb and functions.
5. **Adding of GenAI Connector inside Chat Flyout**: If the user has
admin rights, it is possible to set up a GenAI connector from within the
Observability AI Assistant Chat Flyout. This provides a faster and more
seamless onboarding experience. If the user does not, she will be
redirected to the Settings page.
## Bug fixes
* Fixes chat item styling issues (padding, background color).
## How to test
* Check if the Stack Management plugin works on stateful
* Check if the AI Assistant Settings plugin works on stateful +
serverless
* Check if CRUD operations on KB work
* Check if searching on KB entries work
* Check if its possible to navigate to KB tab directly
(`app/management/kibana/aiAssistantManagementObservability?tab=knowledge_base`)
## Todo
- [x] Add sorting to getEntries
- [x] Add params for tab routing
- [x] Add unit tests
- [ ] Add API tests
- [ ] Add fallback for already indexed entries when searching
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
