## Summary
- Moves the Serverless Endpoint Management tests from
`x-pack/test_serverless/functional/test_suites/security/cypress` to
`x-pack/plugins/security_solution/public/management/cypress` along with
its set of supporting `screens` and `task`'s
- This work is in preparation for enabling tests to be run in a
serverless environment
> ❗❗ NOTE
> Tests for serverless are not currently running. The kibana core team
is working on providing serverless users that have serverless security
roles assigned to them for testing. Also, an effort in underway to also
use a tagging approach to define which existing tests should run in
serverless.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
We try to reuse existing FTR services for serverless test as much as
possible. Most of the services are deployment agnostic, however some
won't work on serverless and we add its own serverless implementation.
Currently we import all the services from api integration tests directly
into the serverless services. The same with functional ones. This PR
loads the minimum required services for api integration and functional
tests respectively.
This way we can control which services are deployment agnostic and
explicitly remove any of it when we decide to have its own
implementation for serverless.
- Move logic to fetch the TopN functions to
'profiling-data-access-plugin'
- Create new TopN functions embeddable
- Refactor Universal profiling page on APM adding Tabs (Flamegraph | Top
10 functions)
- Create a new API on APM to fetch Top functions
<img width="1605" alt="Screenshot 2023-09-08 at 11 14 13"
src="76fc7bf3-094b-438b-99b8-3cab01539eb4">
<img width="1615" alt="Screenshot 2023-09-08 at 11 14 20"
src="c4d3e97a-eee0-4829-8d6f-545f9c844b18">
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Joseph Crail <joseph.crail@elastic.co>
> **Caveat**: the functional flow logic we've adopted for these
components is not one I would encourage, specifically, using "drilled"
`onChange` handlers and utilizing a composing-component-based store.
Ideally, we'd use a `redux` store, or, at the very least a React
`reducer`.
>
> In the interest of time and compatibility, we've opted to use the
pattern from the original components in `advancedSettings`. We plan to
revisit the state management and prop-drilling when `advancedSettings`
is refactored with these components.
> This PR is a prerequisite to
https://github.com/elastic/kibana/pull/166319 and to completing the
Advanced Settings page in Serverless.
> **Note**: Github appears to be a bit confused in the diff comparison,
due to the addition to `React.forwardRef`... apologies for the noise.
## Summary
While working on https://github.com/elastic/kibana/pull/166319 I found a
number of bugs when working against actual UI settings stored in Kibana.
This PR addresses those issues without waiting for the Settings page to
be complete:
- Refactors `input` components to have cleaner APIs, including
`unsavedChange` and `field` "all the way down".
- This cleans up confusing logic, and sets us up for Redux actions.
- Creates a `normalizeSettings` function.
- Settings returned from the `UiSettingsService` in an actual deployment
of Kibana are drastically unpredictable. In some cases, `type` is
missing, but `value` is there... or `value` is missing entirely, but a
`userValue` is there.
- This function "normalizes" the data, deriving missing parts where
possible.
- Changes the `onChangeFn` to accept `undefined` to indicate an unsaved
change has been reverted, rather than relying on the _value_ in the
unsaved change.
- This fixes a number of issues around resets and reverts.
- Alters the `unsavedChange` prop to be undefined, (to indicate the lack
of an unsaved change), rather than an undefined value.
- Fixes an issue where the `ImageFieldInput` wasn't removing a file that
had been set when resetting to default;
- Adds an imperative ref to `FieldInput` components to reset a field's
input beyond resetting the value, (if necessary).
- Fixes the Storybook `common` setups to allow for changes to the
`onChange` types.
- Fixed a bug where the `FieldRow` was indexing an unsaved change by
`name`, rather than by `id`.
- Fixed an issue where the reset link wasn't always clearing a change to
the default value.
- Fixes an issue with the aria label not being derived properly using
the `query`.
- Splits the utility functions into their respective namespaces:
`settings` and `fields`.
- Adds a few more tests to the utility functions to catch logic errors.
## Summary
(this is the continuation of
https://github.com/elastic/kibana/pull/143910, which I started before my
parental leave and which is impossible to rebase)
With the introduction of more features that are part of licenses, we're
also adding more upsells to Kibana. These upsells advertise the feature,
they explain which license is required in order to use said feature and
they will link the client to the subscription page. Take the upsell for
more insights in the alert flyout as an example:
<img width="1584" alt="Screenshot 2022-10-18 at 16 39 52"
src="https://user-images.githubusercontent.com/68591/197629708-17978c8b-595e-4797-b80a-59c799896509.png">
Upsells come in all different shapes. Somtimes they're just links,
sometimes full pages and sometimes interactive popups. They are also
used across all solutions in Kibana.
There is currently no specific tracking in place for these types of
elements yet. Meaning we don't know how many people interact with them,
how many custerms see them and how well they perform in terms of
conversions.
It is technically already possible to analyze clicks on these elements
as part of the regular Kibana click tracking but it would require
setting up queries with lots of `data-test-subj` and `url` filters for
the right click events. Even if we wanted to set up tracking dashboards
with that data, we would still not know how often upsells are seen which
is necessary to calculate their click-through-rate. That rate can give
an indicator if an upsell performs well or if we might want to improve
it in the future.
For that reason, I'm proposing a dedicated set of tracking methods to
capture `impressions` and `clicks` for upsells. No conversion tracking
as of yet, but I will get back to that later.
This PR introduces the `@kbn/subscription-tracking` package. It
leverages the `@kbn/analytics-client` package to send dedicated
subscription tracking events.
It comes with a set of React components that automatically track
impressions and click events. Consumers of those components only need to
specify a `subscription context` that gives more details on the type of
feature that is advertised and the location of the upsell.
```typescript
import { SubscriptionLink } from '@kbn/subscription-tracking';
import type { SubscriptionContextData } from '@kbn/subscription-tracking';
const subscriptionContext: SubscriptionContextData = {
feature: 'threat-intelligence',
source: 'security__threat-intelligence',
};
export const Paywall = () => {
return (
<div>
<SubscriptionLink subscriptionContext={subscriptionContext}>
Upgrade to Platinum to get this feature
</SubscriptionLink>
</div>
)
}
```
The example above uses a `SubscriptionLink` which is a wrapper of
`EuiLink` . So it behaves just like a normal link. Alternatively,
upsells can also use a `SubscriptionButton` or `SubscriptionButtonEmpty`
which wrap `EuiButton` and `EuiButtonEmpty` respectively.
When the link is mounted, it will send off an `impression` event with
the given `subscriptionContext`. That piece of metadata consists of an
identifier of the advertised feature (in this case
`threat-intelligence`) and the `source` of the impression (in this case
the `threat-intelligence` page in the `security` solution). `source`
follows the following format:
`{solution-identifier}__location-identifier`.
There are no special rules for how to name these identifiers but it's
good practise to make sure that `feature` has the same value for all
upsells advertising the same feature (e.g. use enums for features to
prevent spelling mistakes).
Upon interaction with the upsell link/button, a special `click` event is
sent, which, again, contains the same subscription context.
If you want to use the `subscription-tracking` elements in your
solution, you have to set up a `SubscriptionTrackingProvider` in your
plugin setup and register the tracking events on startup. This PR
contains an example for this setup for the security plugin and some of
its sub-plugins.
## Next steps
- There are currently no dedicated tracking dashboards for these events
which I am planning to set up in the future.
- Since I only had a week's worth of time, I did not manage to add
conversion tracking. The addition of those events might be a lot harder
as well since the current license flow does not integrate seamlessly
into Kibana
- All upsells currently link to the license management page which
currently does not inform customers about our license and cloud
offering. It seems to me like a weak link in the subscription funnel and
it would be great to improve on that page.
- potential improvement: Send `impression` event when the element
becomes visible in the viewport instead of when the element is mounted
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/161876
Creates a plugin providing utilities to access metrics data. The plugin
only exposes a server API which includes a client with two methods:
- `getMetricIndices` to retrieve the user-defined indices where metrics
are located
- `updateMetricIndices` to update the indices
The client is now used where we previously relied on infra plugin to
provide the configuration, in APM and Infra.
The plugin persists the configuration in a new saved object
`metrics-data-source`. Because this configuration was previously stored
in the `infrastructure-ui-source`, the plugin relies on a fallback to
reuse any existing value (see additional context
https://github.com/elastic/kibana/issues/161876#issuecomment-1673537400).
### Reviewers
There are no functional changes outside of Infra Monitoring UI and APM
UI, other codeowners are involved because this introduces a new saved
object
- APM - the change introduces a drop-in replacement of the
`infra.getMetricIndices` call. The ui code still relies on infra plugin
for a couple of components so we can't drop the dependency yet, those
we'll need to be moved to a tier 2 plugin (more details in
https://github.com/elastic/observability-dev/discussions/2787
(internal)) in a separate issue
### Testing
You'll need metrics data to verify data fetching works (I've used an
edge-oblt cluster)
1. Navigate to Infrastructure Settings and verify metric indices are
configured with the default value of `infrastructure-ui-source`
2. Update metric indices settings (if connected to oblt cluster add
`remote_cluster:..` indices)
3. Verify `metrics-data-source` saved object is persisted with correct
attributes
4. Verify Infrastructure Inventory is pulling data from the newly
configured indices
5. Go to APM services, verify service Infrastructure pulls data from
newly configured indices
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Jason Rhodes <jason.rhodes@elastic.co>
## Summary
This closes#163788.
## High level overview
- Adds a new `kbn-custom-integrations` package.
- This package adds a new top level custom integrations state machine,
which manages a child create custom integration state machine. In the
future we will have additional modes (such as adding a dataset to an
existing integration, and various "uplift" flows).
- Adds connected (to the machine) components that consumers can use to
facilitate custom integration workflows.
- Adds a `kbn-xstate-utils` package (as these utils were in 2 plugins
and now 1 package).
- Replaces the integration creation inside of the onboarding wizard flow
with this package.
- At the moment this is locked down to `logs`, and one dataset, but it
can be easily extended in the future to support all types and multiple
datasets. The state machine is ready, it just needs exposed in the UI.
- Some thought has gone in to how this will work with multiple "modes",
and the foundations are there (imagining that certain types will be
unions etc), however it's worth not getting too bogged down in those
specific implementation details as I'd rather base that evolution on the
real world usage when we have it.
The Configure integration section should more or less work the same as
before.
## Testing
- When utilising the onboarding flow for custom logs at
`/app/observabilityOnboarding/customLogs` can you:
- Create a custom integration? (It's worth verifying the network
requests, and the assets are installed).
- If you navigate forward, then back, make a change to the integration
fields, and navigate forward again is the previously created integration
deleted?
- Is the success callout with the integration name shown on the next
wizard panel?
- Do field validations work?
- Are errors displayed when you try to create an integration with a name
that already exists?
- Can you retry when there is a server error? (you can block network
requests to the custom integrations API to test this)
## Screenshots
## State machine diagram
(The top level management machine is super basic, so this is just the
create machine)
## Followups
- Tests (the current onboarding UI implementation doesn't have tests so
whilst it's not ideal technically this coverage stays the same)
- Storybook
- Replace other plugins' usage with xstate-utils (not urgent)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Yngrid Coello <yngrid.coello@elastic.co>
## Summary
- Adds support for `apiKey` to the CLI service methods:
`createKbnClient()`, `createEsClient()` and `createRuntimeServices()`
- Note: no existing CLI tools have been changed with this PR to support
`apiKey`. Only adding support to the above service methods so that they
can be used by existing or new CLI utilities
Addresses https://github.com/elastic/kibana/issues/160411
## Summary
This PR adds functionality for filtering out advanced settings that are
not relevant for serverless.
For context, we need to build an Advanced settings page in serverless
which only contains a set of the existing settings. We will reuse the
section registry (https://github.com/elastic/kibana/pull/163502) from
the original Advanced settings plugin as well as its UI components which
will also be extracted into a separate package. The app will be
registered from inside the `serverless` plugin.
In order to only display the settings that are relevant for serverless,
we need to make some changes to the uiSettings service. The
implementation in this PR leverages the existing `readonly` uiSettings
param and adds the `setAllowlist()` method which is called by the
serverless plugin to set an allowlist of setting keys.
**Testing in serverless:**
1. Set `advanced_settings.enabled: true` to enable the Advanced settings
app in serverless:
5b216c6ea9/config/serverless.yml (L53)
2. Start Es with `yarn es serverless --ssl` and Kibana with `yarn
serverless-{mode} --ssl` in any serverless mode.
3. Navigate to `app/management/kibana/settings`
4. Verify that the app only displays the settings from
`packages/serverless/settings/common/index.ts` (these are the settings,
relevant for all projects in serverless) as well as the settings from
the corresponding project package
`packages/serverless/settings/{mode}_project/index.ts`.
5. Verify that the app is functioning correctly.
**Testing in self-managed:**
1. Start Es with `yarn es snapshot` and Kibana with `yarn start`.
2. Go to Stack Management > Advanced settings
3. Verify that all settings are displayed as usual.
4. Verify that the app is functioning correctly.
If your team is a code owner of any of the serverless project plugins,
please review the corresponding package
`packages/serverless/settings/{search/observanility/security}_project/index.ts`
where you've been added as an owner and test in the serverless solution
accordingly.
<!---
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
-->
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
**Resolves: https://github.com/elastic/kibana/issues/164441**
## Summary
Skips all Cypress tests owned by
@elastic/security-detection-rule-management and
@elastic/security-detection-engine teams in Serverless using the new
`@skipInServerless` tag.
- Adds a new `@skipInServerless` tag
- Updates `x-pack/test/security_solution_cypress/cypress/README.md` to
explain when to use what tags
- Explicitly adds missing tags to all D&R tests
- Adds `// TODO:` comments to tests with links to follow-up tickets
- Fixes the
`x-pack/plugins/security_solution/scripts/run_cypress/parallel.ts`
script (see below)
Follow-up work:
- https://github.com/elastic/kibana/issues/161540
- https://github.com/elastic/kibana/issues/161539
## Context
> Serverless test failures will soon block PR merge
> During the development of the serverless test infrastructure, we
decided that serverless tests will only soft-fail. That means you see
the test failure in your PR but you're still able to merge. We did that
mainly in order to not block delivery of stateful features and bug fixes
while serverless tests and pipelines were implemented and stabilized.
We now have the major building blocks for the serverless test
infrastructure in place and will integrate serverless tests in our
regular pipelines. As part of this process, we're skipping failing and
flaky serverless tests that came in during the soft-fail phase. Once
this is done, a PR with serverless test failures can no longer be
merged, similar to how we have it for stateful test failures.
> We plan to merge this in the next few days and we'll send out another
notification when it's done.
## Fixing `parallel.ts`
There were two problems with the
`x-pack/plugins/security_solution/scripts/run_cypress/parallel.ts`
script we use for running Cypress tests:
- The script was broken in https://github.com/elastic/kibana/pull/162673
(here on [this
line](https://github.com/elastic/kibana/pull/162673/files#diff-9f40ced6d29c4fc2709af881680400293d8ce1bc9ebb07b9138d6d99c83c09c9R67))
- I think it has never supported situations when all tests matching a
spec pattern (such as
`./cypress/e2e/!(investigations|explore)/**/*.cy.ts`) end up being
skipped via Cypress tags (such as `@skipInServerless`)
Both the issues are fixed in this PR.
Code owners are added for this script in the CODEOWNERS file to prevent
breaking this script in future PRs.
## Summary
This moves connectors functions to a shared package so Serverless Search
and Enterprise Search can both use it.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Current PR includes the next set of changes:
1. Moved `DiscoverGrid` component to a package `@kbn/unified-data-table`
and added `@elastic/kibana-data-discovery` as code owners.
2. Changed `@kbn/unified-data-table` package naming for data grid
related components and methods to correspond `UnifiedDataTable` instead
of `Discover`.
3. Moved hooks `useColumns` and `useRowHeightsOptions` to a package as
its logic belongs to `UnifiedDataTable`.
4. Renamed `DiscoverGridContext` to `UnifiedDataTableContext`.
5. Extended `UnifiedDataTable` interface and functionality with the next
customization options:
- `renderDocumentView?: (displayedRows:
DataTableRecord[],displayedColumns: string[]) => JSX.Element |
undefined;` - callback to render DocumentView when the document is
expanded
- `configRowHeight?: number;` - optional value for providing
configuration setting for UnifiedDataTable rows height
- `showMultiFields?: boolean;` - optional value for providing
configuration setting for enabling to display the complex fields in the
table. Default is true.
- `maxDocFieldsDisplayed?: number;` - optional value for providing
configuration setting for maximum number of document fields to display
in the table. Default is 50.
- `externalControlColumns?: EuiDataGridControlColumn[];` - optional
value for providing EuiDataGridControlColumn list of the additional
leading control columns. UnifiedDataTable includes two control columns:
Open Details and Select.
<img width="522" alt="Screenshot 2023-08-22 at 2 26 57 PM"
src="d796b9c8-2fef-4bcc-a3c9-9f5cc6349ab9">
- `externalAdditionalControls?: React.ReactNode;` - optional value for
providing the additional controls available in the UnifiedDataTable
toolbar to manage it's records or state. UnifiedDataTable includes
Columns, Sorting and Bulk Actions.
<img width="673" alt="Screenshot 2023-08-22 at 2 40 28 PM"
src="f7ac0c87-5310-49dd-9084-1ce01ca0f366">
- `rowsPerPageOptions?: number[];` - optional list of number type values
to set custom UnifiedDataTable paging options to display the records per
page.
- `renderCustomGridBody?: (args: EuiDataGridCustomBodyProps) =>
React.ReactNode;` - An optional function called to completely customize
and control the rendering of EuiDataGrid's body and cell placement.
<img width="1658" alt="Screenshot 2023-08-22 at 2 50 27 PM"
src="14adc18d-73af-40f5-9859-b3c708e265b1">
- `trailingControlColumns?: EuiDataGridControlColumn[];` - optional list
of the `EuiDataGridControlColumn` type for setting trailing control
columns standard for `EuiDataGrid`.
- `visibleCellActions?: number;` - optional value for a custom number of
the visible cell actions in the table
<img width="497" alt="Screenshot 2023-08-22 at 2 45 49 PM"
src="57ef3ad9-7401-46bb-9b38-cc8cca2e6a24">
- `externalCustomRenderers?: Record<string,(props:
EuiDataGridCellValueElementProps) => React.ReactNode>;` - an optional
settings for a specified fields rendering like links. Applied only for
the listed fields rendering:
<img width="1121" alt="Screenshot 2023-08-22 at 2 51 07 PM"
src="77501eae-3046-4a2c-90e1-2db487c21e2c">
- `consumer` - optional string value for the name of the
`UnifiedDataTable` consumer component or application.
6. Extended `UnifiedDataGrid` services with the two additional:
`storage: Storage;`
`data: DataPublicPluginStart; `
replaced `core: CoreStart;` with `theme: ThemeServiceStart;`, because
`core` is used only to get `theme`
7. Replaced `DocumentView` property with `renderDocumentView?:
(displayedRows: DataTableRecord[],displayedColumns: string[]) =>
JSX.Element | undefined;` callback function, which allows not to use
`DiscoverGridFlyout` definition for the documents rendering.
```
/**
* Document detail view component
*/
DocumentView?: typeof DiscoverGridFlyout;
```
8. Removed the next properties from the data table interface, because it
was used to render DiscoverGridFlyout:
```
/**
* Filters applied by saved search embeddable
*/
filters?: Filter[];
/**
* Query applied by KQL bar or text based editor
*/
query?: Query | AggregateQuery;
/**
* Saved search id used for links to single doc and surrounding docs in the flyout
*/
savedSearchId?: string;
```
9. Added usage examples and interface description to README file.
10. Changed grid styles naming from `.dscDiscoverGrid*` to
`.udtDataTable*`
11. Migrated discover plugin to use `UnifiedDataTable` instead of
`DiscoverGrid`
Extra changes were needed to avoid the circular dependancies:
- moved `DocViewFilterFn` and `FieldMapping` from discover plugin to
`packages/kbn-discover-utils/src/types.ts`
- added own `export type SortOrder = [string, string];` to avoid deps
for saved-search plugin
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This is part 1 of a series of PRs to expose the flamegraph to be used by
other plugins.
**The problem**
Currently for plugin-A to show data from plugin-B, it needs to add
dependency on plugin-B. If plugin-B wants to show data from plugin-A, it
also needs to add plugin-A as a dependency, and here is where the
problem happens. In such scenario, we have a cyclic dependency problem.
**The solution**
To solve this problem a new plugin is created, `profiling-data-access`.
This plugin exposes services that consumer plugins can call in other to
have the data they need to show on their end. The `profiling` plugin is
also using this new plugin. For now, only the flamegraph service is
available, The idea is to slowly migrate the data fetching from
profiling to this new plugin in other to facilitate the integration
across plugins.
**Why some many files?**
As I said, only the flamegraph logic was moved to the new plugin, but it
has many files that it needs to properly build the response of the
service call. I moved all these files to the `common` folder inside the
new plugin and adjusted the imports in the profiling plugin.
<img width="1032" alt="Screenshot 2023-08-31 at 09 29 27"
src="287bd28e-b834-45e0-8042-576d1fcff6cd">
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
### What this PR changes
branched from elastic/kibana/pull/163759
- Introduces new AppFeatures package `@kbn/security-solution-features`
with the common logic and `AppFeatureService` to apply offering specific
configurations for Security Solution features independently for
Serverless and ESS. This logic is replacing the earlier `AppFeatures` in
order to introduce new Kibana feature privileges for serverless PLIs so
that new Kibana privileges introduced for serverless PLIs do not
affect/show up as new Kibana feature privileges in ESS.
- Gates endpoint exceptions on alerts/rules based on serverless PLI
configurations. On serverless `Endpoint exceptions` should be
accessible/seen only on endpoint essentials/complete.
New AppFeatures logic architecture diagram:
**Note:** Corresponding API changes related to endpoint exceptions will
be in a new PR, along with the last set of UX changes for hiding the
`Endpoint exceptions` tab from the Rules details page.
### How to review
- Setup for _Servlerless_
- Run `yarn es snapshot` on a terminal window to start ES.
- Copy `config/serverless.security.yml` to
`config/serverless.security.dev.yml`
- Run `yarn serverless-security --no-base-path` on another terminal
window to start kibana in serverless mode
- Run `node
x-pack/plugins/security_solution/scripts/endpoint/endpoint_agent_emulator.js
--asSuperuser` on a new window and then select `1` to `Load Endoints`
and then `1` to `Run` the loader script. This will load some fake
agents/alerts data to test with.
### Tests (Serverless)
- with
`{ product_line: 'security', product_tier: 'essentials' }` or `{
product_line: 'security', product_tier: 'complete' }`
and
`{ product_line: 'endpoint', product_tier: 'essentials' }` or `{
product_line: 'endpoint', product_tier: 'complete' }`
1. Navigate to Rules>Shared exception lists via
`http://localhost:5601/app/security/exceptions`
2. Test that you can see `Endpoint Security Exception List` card on the
shared exception lists page.
3. Navigate to `Alerts` page via `app/security/alerts`, you should see
endpoint alerts. If not, then click on `Manage Rules` and then
disable/enable `Endpoint Security` rules. That should trigger alerts to
show up on the Alerts table.
4. Click on `View Details` button under `Actions` column. Once the
flyout is visible, click on `Take Action` and verify that `Add Endpoint
exception` is visible/enabled/clickable on the menu.
5. Click on `More actions` button under `Actions` column and verify that
`Add Endpoint exception` is visible/enabled/clickable on the menu.
6. Click on `Investigate in timeline` button under `Actions` column;
when the timeline view is visible and the alert item is displayed, click
on buttons mentioned in 4. and 5. above and verify the same.
7. Navigate to `Rules`>`DetectionRules`>`Endpoint Security` rule under
the `Rules` table. Select the `Alerts` tab.
8. Click and verify `View details`,`More actions` and `Investigate in
timeline` buttons same as in 4., 5., 6. above.
9. You should be able to see the `Endpoint exceptions` tab as well.
Click and verify that you can see the tab's content.
- with
`{ product_line: 'security', product_tier: 'essentials' }` or `{
product_line: 'security', product_tier: 'complete' }`
1. Edit `config/serverless.security.dev.yml` so that `endpoint` product
line item is commented out.
2. Test that you can not see `Endpoint Security Exception List` card on
the shared exception lists page.
3. Items 4. 5. 6. as above but the menu items should be disabled. This
can be verified with fake data only as with a real endpoint, endpoint
alerts are actually not visible at all.
### Tests (ESS)
On the ESS side, endpoint exceptions are not affected by this change and
work as usual based on index privileges.
---------
Co-authored-by: semd <sergi.massaneda@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: YulNaumenko <jo.naumenko@gmail.com>
Co-authored-by: Pablo Neves Machado <pablo.nevesmachado@elastic.co>
Co-authored-by: Pablo Machado <machadoum@gmail.com>
## [Security Solution] [Elastic AI Assistant] LangChain integration (experimental)
This PR integrates [LangChain](https://www.langchain.com/) with the [Elastic AI Assistant](https://www.elastic.co/blog/introducing-elastic-ai-assistant) as an experimental, alternative execution path.
### How it works
- There are virtually no client side changes to the assistant, apart from a new branch in `x-pack/packages/kbn-elastic-assistant/impl/assistant/api.tsx` that chooses a path based on the value of the `assistantLangChain` flag:
```typescript
const path = assistantLangChain
? `/internal/elastic_assistant/actions/connector/${apiConfig?.connectorId}/_execute`
: `/api/actions/connector/${apiConfig?.connectorId}/_execute`;
```
Execution of the LangChain chain happens server-side. The new route still executes the request via the `connectorId` in the route, but the connector won't execute the request exactly as it was sent by the client. Instead, the connector will execute one (or more) prompts that are generated by LangChain.
Requests routed to `/internal/elastic_assistant/actions/connector/${apiConfig?.connectorId}/_execute` will be processed by a new Kibana plugin located in:
```
x-pack/plugins/elastic_assistant
```
- Requests are processed in the `postActionsConnectorExecuteRoute` handler in `x-pack/plugins/elastic_assistant/server/routes/post_actions_connector_execute.ts`.
The `postActionsConnectorExecuteRoute` route handler:
1. Extracts the chat messages sent by the assistant
2. Converts the extracted messages to the format expected by LangChain
3. Passes the converted messages to `executeCustomLlmChain`
- The `executeCustomLlmChain` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/execute_custom_llm_chain/index.ts`:
1. Splits the messages into `pastMessages` and `latestMessage`, where the latter contains only the last message sent by the user
2. Wraps the conversation history in the `BufferMemory` LangChain abstraction
3. Executes the chain, kicking it off with `latestMessage`
```typescript
const llm = new ActionsClientLlm({ actions, connectorId, request });
const pastMessages = langchainMessages.slice(0, -1); // all but the last message
const latestMessage = langchainMessages.slice(-1); // the last message
const memory = new BufferMemory({
chatHistory: new ChatMessageHistory(pastMessages),
});
const chain = new ConversationChain({ llm, memory });
await chain.call({ input: latestMessage[0].content }); // kick off the chain with the last message
};
```
- When LangChain executes the chain, it will invoke `ActionsClientLlm`'s `_call` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/llm/actions_client_llm.ts` one or more times.
The `_call` function's signature is defined by LangChain:
```
async _call(prompt: string): Promise<string>
```
- The contents of `prompt` are completely determined by LangChain.
- The string returned by the promise is the "answer" from the LLM
The `ActionsClientLlm` extends LangChain's LLM interface:
```typescript
export class ActionsClientLlm extends LLM
```
This let's us do additional "work" in the `_call` function:
1. Create a new assistant message using the contents of the `prompt` (`string`) argument to `_call`
2. Create a request body in the format expected by the connector
3. Create an actions client from the authenticated request context
4. Execute the actions client with the request body
5. Save the raw response from the connector, because that's what the assistant expects
6. Return the result as a plain string, as per the contact of `_call`
## Desk testing
This experimental LangChain integration may NOT be enabled via a feature flag (yet).
Set
```typescript
assistantLangChain={true}
```
in `x-pack/plugins/security_solution/public/app/app.tsx` to enable this experimental feature in development environments.
UX: https://github.com/elastic/security-team/issues/7310
## Summary
* It creates an Upselling package to share the service and components
between ESS and Serverless plugins
* It implements upselling for entity analytics on serverless by
replicating the ESS approach
ESS
<img width="1520" alt="Screenshot 2023-08-17 at 13 34 59"
src="95c2c94e-7ab3-4d9f-aa24-b3e9c00eb3ed">
Serverless
<img width="1523" alt="Screenshot 2023-08-17 at 13 39 25"
src="618ce9dc-ef4e-469d-884a-dfb09834d0b0">
We are not displaying the upgrade button because we still don't know how
to link to the cloud settings page.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This fixes some missed spots when I changed CODEOWNERS in this PR:
https://github.com/elastic/kibana/pull/164122
Forgot to update all the kibana.jsonc files, so CI overwrote my changes
in CODEOWNERS :)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Allow Kibana to restrict the usage of JWT for a predefined set of routes
only in Serverless environment by default. This capability is not
available in non-Serverless environment.
Any route that needs to be accessed in Serverless environemnt using JWT
as a means of authentication should include `security:acceptJWT` tag.
## How to test
If you'd like to generate your own JWT to test the PR, please follow the
steps outlined in
https://github.com/elastic/kibana/pull/159117#issue-1743796706 or just
run functional test server and use static JWT from the Serverless test.
This PR also generated a Serverless Docker image that you can use in
your Dev/QA MKI cluster.
- [x] Implementation functionality and add unit tests
- [x] Update metrics/status routes to include new `security:acceptJWT`
tag
- [x] Update serverless test suite to include a test for
`security:acceptJWT`
__Fixes: https://github.com/elastic/kibana/issues/162632__
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
