## Summary
close https://github.com/elastic/kibana/issues/168696
- Remove old copy from `kibana_react`
- Remove -types and -mocks packages, move them into a single package
(not needed since we don't use bazel now)
### Risks
No Risks, since old copy was not used
**Parent meta ticket:
https://github.com/elastic/security-team/issues/7491**
Resolves: https://github.com/elastic/security-team/issues/7582
Resolves: https://github.com/elastic/security-team/issues/7580
Resolves: https://github.com/elastic/security-team/issues/7581
## Summary
This PR migrates the rules schema to OpenAPI, Zod, and code generation.
The following APIs now have complete OpenAPI specifications and are
enabled for code generation:
| Method | Endpoint | OpenAPI spec | Fully migrated |
| ------ |
---------------------------------------------------------------- |
------------ | -------------- |
| POST | /api/detection_engine/rules | ✅ | ✅ |
| GET | /api/detection_engine/rules | ✅ | ✅ |
| PUT | /api/detection_engine/rules | ✅ | ✅ |
| PATCH | /api/detection_engine/rules | ✅ | ✅ |
| DELETE | /api/detection_engine/rules | ✅ | ✅ |
| POST | /api/detection_engine/rules/\_bulk_create | ✅ | ✅ |
| PUT | /api/detection_engine/rules/\_bulk_update | ✅ | ✅ |
| PATCH | /api/detection_engine/rules/\_bulk_update | ✅ | ✅ |
| DELETE | /api/detection_engine/rules/\_bulk_delete | ✅ | ✅ |
| POST | /api/detection_engine/rules/\_bulk_delete | ✅ | ✅ |
### Rule schemas are now forward-compatible
We now allow extra fields in schemas for forward compatibility, but we
remove them from the payload during parsing. So from now on, extra
fields are simply ignored and won't lead to validation errors.
## 📓 Summary
Closes#169394
This PR extends the `flyout` customization extension point to support
updating/replacing the content shown in the document flyout.
A consumer would need to show/hide/highlight details related to the
expanded document, and it might also want to control whether the default
content (currently only the UnifiedDocViewer) is shown/hidden. Finally,
it could be necessary to perform imperative actions such as
adding/removing columns or filter.
To get this flexibility at the moment of customizing the content, the
existing extension point takes now a `Content` component, where some
props are injected.
```
export interface FlyoutContentProps {
actions: {
setFilter?: DocViewFilterFn;
addColumn: (column: string) => void;
removeColumn: (column: string) => void;
};
doc: DataTableRecord;
renderDefaultContent: () => React.ReactNode;
}
```
N.B. `renderDefaultContent` is passed as a function instead of a React
element to avoid its creation in the Discover flyout in case the
consumer doesn't want to display it.
Here is a usage example of the new extension point property.
```
customizations.set({
id: 'flyout',
Content: ({ actions, doc, renderDefaultContent }) => {
return (
<Panel>
<HighlightComponent timestamp={doc.flattened['@timestamp']} />
<Columns onAddColumn={actions.addColumns} onAddColumn={actions.removeColumn} />
<Filters onFilter={actions.setFilter} />
{renderDefaultContent()}
</Panel>
);
},
});
```
---------
Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
### Summary
Fix https://github.com/elastic/kibana/issues/166688
Implements dynamic contract resolving for plugins, allowing to retrieve
contracts after their respective lifecycle is completed, and therefore
working around cyclic dependencies.
In term of workflow execution, we're basically going from
<img width="842" alt="Screenshot 2023-09-27 at 08 09 27"
src="251637d1-ec97-4071-a445-2f59512ce187">
to:
<img width="1092" alt="Screenshot 2023-09-27 at 08 09 32"
src="de466cda-7e43-4fd3-81ec-4339d05d279d">
### API
This functionality is exposed by the now publicly exposed `plugins`
service contracts:
```ts
setup(core) {
core.plugins.onSetup<{pluginA: SetupContractA, pluginB: SetupContractA}>('pluginA', 'pluginB')
.then(({ pluginA, pluginB }) => {
if(pluginA.found && pluginB.found) {
// do something with pluginA.contract and pluginB.contract
}
});
}
```
```ts
start(core) {
core.plugins.onStart<{pluginA: StartContractA, pluginB: StartContractA}>('pluginA', 'pluginB')
.then(({ pluginA, pluginB }) => {
if(pluginA.found && pluginB.found) {
// do something with pluginA.contract and pluginB.contract
}
});
}
```
**remark:** the `setup` contract exposed both `onSetup` and `onStart`,
while the `start` contract only exposed `onStart`. The intent is to
avoid fully disrupting the concept of lifecycle stages.
### Guardrails
To prevent developer from abusing this new API, or at least to add some
visibility on its adoption, plugins can only perforn dynamic contract
resolving against dependencies explicitly defined in their manifest:
- any required dependencies (*existing concept*)
- any optional dependencies (*existing concept*)
- any runtime dependencies (**new concept**)
Runtime dependencies must be specified using the new
`runtimePluginDependencies` field of a plugin's manifest.
```json
{
"type": "plugin",
"id": "@kbn/some-id",
"owner": "@elastic/kibana-core",
"plugin": {
"id": "some-id",
"...": "...",
"runtimePluginDependencies" : ["someOtherPluginId"]
}
}
```
Using the contract resolving API will throw at call time when trying to
resolve the contract for an undeclared dependency.
E.g this would throw at invocation time (not returning a rejected
promise - throw).
```ts
setup(core) {
core.plugins.onSetup<{undeclaredDependency: SomeContract}>('undeclaredDependency');
}
```
The reasoning behind throwing is that these errors should only occur
during the development process, and an hard fail is way more visible
than a promise rejection that should be more easily shallowed.
### Code reviews
This PR defines @elastic/kibana-core as codeowner of all `kibana.jsonc`
files in the `src/plugins` and `x-pack/plugins` directories, so that a
code review will be triggered whenever anyone changes something in any
manifest. The intent is to be able to monitor new usages of the feature,
via the addition of entries in the `runtimePluginDependencies` option of
the manifest.
### Remarks
Exposing this API, and therefore making possible cyclic dependencies
between plugins, opens the door to other questions.
For instance, cross-plugin type imports are not technically possible at
the moment, given that plugins are referencing each others via TS refs,
and refs forbid cyclic dependencies. Which means that to leverage this
to address cyclic dependency issues, the public types of **at least one
of the two** plugins will have to be extracted to a shared place (likely
a package).
Resolving, or trying to improve the developer experience around this
issue, is absolutely out of scope of the current PR (and the issue it
addresses).
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
- Resolves https://github.com/elastic/kibana/issues/166906
## Summary
This PR adds field tokens to column header for the grid.
There are also additional necessary changes:
- field utils (name, description, icon) were moved from
`@kbn/unified-field-list` and `@kbn/discover-utils` to its own new
package `@kbn/field-utils`
- Unified Data Table and Unified Doc Viewer got a new prop `columnTypes`
which allows to render correct field icons for ES|QL searches (before
types were derived from the data view fields which could be misleading
in text-based searches as users can customize field names via query)
<img width="600" alt="Screenshot 2023-09-25 at 19 30 21"
src="388de9bb-94f7-4d3e-878a-ca0da99fcec2">
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [x] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Davis McPhee <davis.mcphee@elastic.co>
## Summary
Create utilities allowing to simulate a model version cohabitation
period, by creating two SO repositories of different versions so that
type/test owners can use it to assert the behavior of documents created
by one version and read by the other, or vice-versa.
### Usage example
```ts
describe('myIntegrationTest', () => {
const testbed = createModelVersionTestBed();
let testkit: ModelVersionTestKit;
beforeAll(async () => {
await testbed.startES();
});
afterAll(async () => {
await testbed.stopES();
});
beforeEach(async () => {
testkit = await testbed.prepareTestKit({
savedObjectDefinitions: [{
definition: mySoTypeDefinition,
modelVersionBefore: 1,
modelVersionAfter: 2,
}]
})
});
afterEach(async () => {
if(testkit) {
await testkit.tearsDown();
}
});
it('can be used to test model version cohabitation', async () => {
// last registered version is `1`
const repositoryV1 = testkit.repositoryBefore;
// last registered version is `2`
const repositoryV2 = testkit.repositoryAfter;
// do something with the two repositories, e.g
await repositoryV1.create(someAttrs, { id });
const v2docReadFromV1 = await repositoryV2.get('my-type', id);
expect(v2docReadFromV1.attributes).toEqual(something);
})
})
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Close https://github.com/elastic/kibana/issues/159691
[Requirements](https://docs.google.com/document/d/1uXgyDIGuIqkYXmavdMTpBgQEiOP07ObJYb7GNq5GiSE/edit#heading=h.okl11rz12ytg)
[A/B test
description](https://docs.google.com/document/d/1yzfZF8mtlRNH4X6HjosD6Exh24zAhN__LR_zh3DOzKw/edit?usp=sharing)
[Figma](https://www.figma.com/file/WGhmfgyy9FBOltLtycfGPE/Getting-started?type=design&node-id=92-44804&mode=design&t=mwbIexn5Fs754HQz-0)
Testing - see _testing_ section below for more details
(https://dosant-pr-167374-d-2023-06-14-global-drift-with-experiment.kbndev.co/.
elastic/changeme)
This PR enables cloud chat (Drift) globally. This is done by adding a
custom chat button in the Kibana header which manually toggles Drift
widget. We attempt to manually position the widget to the top of the
screen so it pops up close to the chat button that triggered it.
Previously Drift chat was available only on specific pages like
Solutions onboarding pages, integrations, setup guides as a regular chat
widget with the floating chat bubble in the bottom right corner. We
couldn't enable it on all pages, because on a lot of them the floating
chat bottom would have overlapped the application UI.
We also were asked to add [an a/b
test](https://docs.google.com/document/d/1yzfZF8mtlRNH4X6HjosD6Exh24zAhN__LR_zh3DOzKw/edit?usp=sharing):
- A: The chat button appears in the header for all pages (new)
- B: The chat button appears as floating action button in the
bottom-right corner on pages where Drift was previously available
(solutions onboarding pages, integrations, setup guides)
### Screenshots / Videos
#### Global Chat in the header
#### The tour on the first appearance
#### (Part of A/B test) Drift in the header on new pages and as floating
action button on old pages
0386ccbd-ab6c-4eb2-a57b-f9324fcf73eb
### Implementation notes
- **We still enable Drift only for trial users + gap window**
- We exposed additional APIs from drift iframe to manually control its
visibility and react to more events
https://github.com/elastic/cloud/pull/118761. This changes are required
for the code in this PR to work. ~The updated frame code wasn't deployed
yet.~ the changes were deployed
- We use [`playbookFired` event
](https://devdocs.drift.com/docs/drift-events#playbook-fired) to know if
Drift chat should be visible for the current user. We show the button in
the header only when it fires.
- To react to the event and to display the button, we have to kick of
Drift iframe initialization first
- This means Drift codes loads before we show the button and before user
interacts with it (Only when Drift is enabled, meaning, only for trial
users + gap window)
- Subsequent launches or opens of the same playbook will not re-trigger
the `playbookFired` event, I used local storage flag to workaround this
and show the live chat button, but it also has it's own edge case. As an
alternative we can always show the chat button and don't rely on the
playbook event, more details here:
https://docs.google.com/document/d/1j313mVOIz19Rkoj8TDFWaLc7Pgk_ByBBKJFNIC-jbyc/edit?usp=sharing.
For now was decided to rely on the `playbookFired` event.
- A/B: **to support for both new and old implementation, I had to
refactor the old one from drop-in chat to global chat that is controlled
by list of hardcoded URLs.** This is not ideal, but this allows two
implementations to co-exist with much tech-debt and we plan to get rid
of this after the a/b test.
- When we navigate between pages with different implementations, Drift
re-initializes itself (just like in old implementation), this
performance debt should go away when we get rid of the a/b test.
- When end-to-end testing the a/b experiment, I found a bug in the a/b
test setup in Kibana https://github.com/elastic/kibana/issues/167240
this needs to be addressed separately for a/b test to work properly.
- When the user receives a message from Drift - the custom "Live Chat"
button doesn't indicate that there is a new message. This needs a follow
up, @Dosant to create an issue
### Testing
Version with the experiment where on old pages Drift appears as a
floating chat bubble -
https://dosant-pr-167374-d-2023-06-14-global-drift-with-experiment.kbndev.co/
elastic/changeme
> [!NOTE]
> If the live chat button doesn't appear, it is likely because the
playbook was recently activated by someone else. you can workaround this
for testing by creating a different user or by setting
`cloudChatPlaybookFiredOnce : true` to localstorage. This issue and
mitigation is described in details in the "implementation details"
section
#### To test locally:
```
xpack.cloud.id: 'some-id'
xpack.cloud.trial_end_date: '2023-09-21T00:00:00.000Z'
xpack.cloud_integrations.chat.trialBuffer: 45
xpack.cloud.chat.enabled: true
xpack.cloud.chatIdentitySecret: <pls react out>
xpack.cloud.chat.chatURL: https://elasticcloud-production-chat-us-east-1.s3.amazonaws.com/drift-iframe.html
xpack.cloud_integrations.experiments.flag_overrides:
"cloud-chat.chat-variant": "bubble" or "header"
```
## Summary
This PR adds a Settings application component for rendering the Advanced
Settings page in serverless.
### How to test:
1. Start Es with `yarn es serverless` and Kibana with `yarn
serverless-{es/oblt/security}`
2. Go to Management -> Advanced Settings
3. Verify that the settings can be changed and saved.
### Advanced Settings page:
<img width="1495" alt="Screenshot 2023-09-28 at 20 56 25"
src="374b3bbd-7bf6-4de7-8129-8b293dd1698e">
### Added an Advanced Settings card to the Management landing page:
<img width="1575" alt="Screenshot 2023-09-28 at 12 24 23"
src="c08b8b36-ff40-4772-87d6-597629d78342">
<!--- ### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
-->
---------
Co-authored-by: Clint Andrew Hall <clint@clintandrewhall.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Clint Andrew Hall <clint.hall@elastic.co>
## Summary
This PR wraps up the work the @elastic/kibana-presentation team has done
to finish the MVP of [Phase
1](https://github.com/elastic/kibana/issues/154354) of the `Link`
embeddable, which enables users to add panels to their dashboard that
contain links to other dashboards + external links - with respect to
dashboard links, we give the author control over which pieces of context
should be kept across dashboards so that things like filter pills,
queries, and time ranges are not lost. This marks a huge improvement in
dashboard navigation overall, which was previously only available via a
variety of different workarounds including (but not limited to):
- Creating (essentially) a `noop` dashboard-to-dashboard drilldown
- Using markdown panels with hard Dashboard links, which are prone to
break across updates
- Avoiding navigation all together, which resulted in large,
slow-to-load dashboards.
As an added benefit, because these panels contain **references** to each
dashboard rather than hard links, (1) unlike markdown links, they should
not break after updates and (2) if a links panel is exported and
imported into another space or instance, all of the dashboards it links
to will also be imported.
1a86b713-47e7-4db9-8a04-29d41b13681a
> **Note**
> 🔉 The above video has audio! Turn on your sound for the best
experience.
### Note about this PR
- A majority of this work was done on a feature branch, with thorough
reviews from @andreadelrio on behalf of @elastic/kibana-design along the
way. Therefore, while feedback on the design is encouraged, any large
concerns brought up in this PR should be filed as separate issues and
addressed in follow-up PRs.
- This PR contains work for giving embeddables control over their own
panel size / default positioning on the dashboard. This was especially
important for the links panel, since we assume that (a) most links
panels would be located somewhere near the top of the dashboard and (b)
the horizontal links panel should have a different default "shape"
(longer than it is tall) than the vertical panel (taller than it is
long).
- This PR also contains work for caching dashboard saved objects, which
makes navigation much more seamless.
### Flaky Test Runner
-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3251
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
~[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials~ This will
be addressed in a follow up:
https://github.com/elastic/kibana/issues/166750
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios - ~Units tests
are added, functional tests are forthcoming~ Edit: All tests are in.
- [x] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [x] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [x] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [x] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: Nick Peihl <nick.peihl@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Andrea Del Rio <delrio.andre@gmail.com>
Co-authored-by: Devon Thomson <devon.thomson@elastic.co>
Co-authored-by: Nick Peihl <nickpeihl@gmail.com>
Co-authored-by: Gerard Soldevila <gerard.soldevila@elastic.co>
## Summary
This PR follows #166460 by adding Category panels to the Form.
<img width="1807" alt="Screenshot 2023-09-27 at 3 36 16 PM"
src="2abe8cf5-5822-473f-affd-148fb7949316">
## Notes
This PR is divided into several commits, the first few being
prerequisite codemods. I recommend reviewing each commit separately, as
the codemods might obscure the actual component work.
- [e78586f - Make SettingType pre-defined to clean up
references](e78586fe44)
- This makes the `SettingType` optional, to clean up areas where the
generic need not be specific.
- [80a9988 - [codemod] Make onFieldChange and onInputChange more
distinct](80a9988516)
- The `onChange` handlers weren't very clear as you work your way up the
component tree. This makes the implementation and usage easier to
understand, (and easier to [replace with state
management](https://github.com/elastic/kibana/issues/166579)).
- [5d0beff - [fix] Fix logged errors in form
tests](5d0beff00c)
- This fixes some logged errors in the Form from `Monaco` and from some
missing `act` and `waitFor` calls.
The `chi2test` utils so fare were only used within data comparison view.
We plan to use it with other plugins, so moving it so a separate package
in this PR. `SIGNIFICANCE_LEVELS` was updated to include some more
digits.
## Summary
Closes#162263
Introduces a new `Feature Privileges` section to the developer
documentation. The documentation includes a tutorial on how to control
access to features of plugin being developed. Introduces a few sections:
- Controlling access to UI features
- Controlling access to server side APIs
- Documentation for configuration options
## Testing
To build this locally, run ./scripts/dev_docs from a local checkout of
this PR. A server will eventually start on http://localhost:3000 where
you can preview the changes.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Addresses https://github.com/elastic/kibana/issues/160411
## Summary
This PR adds a package that contains a form component for the Advanced
Settings UI in serverless.
This implementation was extracted from the the `Form` component in the
`advancedSettings` plugin, excluding some functionalities:
- The form doesn't support search queries.
- The form doesn't divide the settings into categories.
### Testing
The form can be tested in the Storybook Preview from the CI build. Some
things to be tested:
- Making changes to any of the fields displays the bottom bar.
- Clicking the Cancel button clears the changes.
- Clicking the Save button triggers a `saveChanges` action with the
correct changes.
- The bottom bar correctly shows the number of unsaved settings.
- Toggling the `isSavingEnabled` control to `false` disables all fields.
- Toggling the `requirePageReload` control to `true` causes saving of
changes to any of the fields to trigger a `showReloadPagePrompt` action.
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [x] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [x] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [x] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
<!--
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
-->
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Clint Andrew Hall <clint@clintandrewhall.com>
## Summary
(this is the continuation of
https://github.com/elastic/kibana/pull/143910, which I started before my
parental leave and which is impossible to rebase)
With the introduction of more features that are part of licenses, we're
also adding more upsells to Kibana. These upsells advertise the feature,
they explain which license is required in order to use said feature and
they will link the client to the subscription page. Take the upsell for
more insights in the alert flyout as an example:
<img width="1584" alt="Screenshot 2022-10-18 at 16 39 52"
src="https://user-images.githubusercontent.com/68591/197629708-17978c8b-595e-4797-b80a-59c799896509.png">
Upsells come in all different shapes. Somtimes they're just links,
sometimes full pages and sometimes interactive popups. They are also
used across all solutions in Kibana.
There is currently no specific tracking in place for these types of
elements yet. Meaning we don't know how many people interact with them,
how many custerms see them and how well they perform in terms of
conversions.
It is technically already possible to analyze clicks on these elements
as part of the regular Kibana click tracking but it would require
setting up queries with lots of `data-test-subj` and `url` filters for
the right click events. Even if we wanted to set up tracking dashboards
with that data, we would still not know how often upsells are seen which
is necessary to calculate their click-through-rate. That rate can give
an indicator if an upsell performs well or if we might want to improve
it in the future.
For that reason, I'm proposing a dedicated set of tracking methods to
capture `impressions` and `clicks` for upsells. No conversion tracking
as of yet, but I will get back to that later.
This PR introduces the `@kbn/subscription-tracking` package. It
leverages the `@kbn/analytics-client` package to send dedicated
subscription tracking events.
It comes with a set of React components that automatically track
impressions and click events. Consumers of those components only need to
specify a `subscription context` that gives more details on the type of
feature that is advertised and the location of the upsell.
```typescript
import { SubscriptionLink } from '@kbn/subscription-tracking';
import type { SubscriptionContextData } from '@kbn/subscription-tracking';
const subscriptionContext: SubscriptionContextData = {
feature: 'threat-intelligence',
source: 'security__threat-intelligence',
};
export const Paywall = () => {
return (
<div>
<SubscriptionLink subscriptionContext={subscriptionContext}>
Upgrade to Platinum to get this feature
</SubscriptionLink>
</div>
)
}
```
The example above uses a `SubscriptionLink` which is a wrapper of
`EuiLink` . So it behaves just like a normal link. Alternatively,
upsells can also use a `SubscriptionButton` or `SubscriptionButtonEmpty`
which wrap `EuiButton` and `EuiButtonEmpty` respectively.
When the link is mounted, it will send off an `impression` event with
the given `subscriptionContext`. That piece of metadata consists of an
identifier of the advertised feature (in this case
`threat-intelligence`) and the `source` of the impression (in this case
the `threat-intelligence` page in the `security` solution). `source`
follows the following format:
`{solution-identifier}__location-identifier`.
There are no special rules for how to name these identifiers but it's
good practise to make sure that `feature` has the same value for all
upsells advertising the same feature (e.g. use enums for features to
prevent spelling mistakes).
Upon interaction with the upsell link/button, a special `click` event is
sent, which, again, contains the same subscription context.
If you want to use the `subscription-tracking` elements in your
solution, you have to set up a `SubscriptionTrackingProvider` in your
plugin setup and register the tracking events on startup. This PR
contains an example for this setup for the security plugin and some of
its sub-plugins.
## Next steps
- There are currently no dedicated tracking dashboards for these events
which I am planning to set up in the future.
- Since I only had a week's worth of time, I did not manage to add
conversion tracking. The addition of those events might be a lot harder
as well since the current license flow does not integrate seamlessly
into Kibana
- All upsells currently link to the license management page which
currently does not inform customers about our license and cloud
offering. It seems to me like a weak link in the subscription funnel and
it would be great to improve on that page.
- potential improvement: Send `impression` event when the element
becomes visible in the viewport instead of when the element is mounted
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/161876
Creates a plugin providing utilities to access metrics data. The plugin
only exposes a server API which includes a client with two methods:
- `getMetricIndices` to retrieve the user-defined indices where metrics
are located
- `updateMetricIndices` to update the indices
The client is now used where we previously relied on infra plugin to
provide the configuration, in APM and Infra.
The plugin persists the configuration in a new saved object
`metrics-data-source`. Because this configuration was previously stored
in the `infrastructure-ui-source`, the plugin relies on a fallback to
reuse any existing value (see additional context
https://github.com/elastic/kibana/issues/161876#issuecomment-1673537400).
### Reviewers
There are no functional changes outside of Infra Monitoring UI and APM
UI, other codeowners are involved because this introduces a new saved
object
- APM - the change introduces a drop-in replacement of the
`infra.getMetricIndices` call. The ui code still relies on infra plugin
for a couple of components so we can't drop the dependency yet, those
we'll need to be moved to a tier 2 plugin (more details in
https://github.com/elastic/observability-dev/discussions/2787
(internal)) in a separate issue
### Testing
You'll need metrics data to verify data fetching works (I've used an
edge-oblt cluster)
1. Navigate to Infrastructure Settings and verify metric indices are
configured with the default value of `infrastructure-ui-source`
2. Update metric indices settings (if connected to oblt cluster add
`remote_cluster:..` indices)
3. Verify `metrics-data-source` saved object is persisted with correct
attributes
4. Verify Infrastructure Inventory is pulling data from the newly
configured indices
5. Go to APM services, verify service Infrastructure pulls data from
newly configured indices
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Jason Rhodes <jason.rhodes@elastic.co>
## Summary
This closes#163788.
## High level overview
- Adds a new `kbn-custom-integrations` package.
- This package adds a new top level custom integrations state machine,
which manages a child create custom integration state machine. In the
future we will have additional modes (such as adding a dataset to an
existing integration, and various "uplift" flows).
- Adds connected (to the machine) components that consumers can use to
facilitate custom integration workflows.
- Adds a `kbn-xstate-utils` package (as these utils were in 2 plugins
and now 1 package).
- Replaces the integration creation inside of the onboarding wizard flow
with this package.
- At the moment this is locked down to `logs`, and one dataset, but it
can be easily extended in the future to support all types and multiple
datasets. The state machine is ready, it just needs exposed in the UI.
- Some thought has gone in to how this will work with multiple "modes",
and the foundations are there (imagining that certain types will be
unions etc), however it's worth not getting too bogged down in those
specific implementation details as I'd rather base that evolution on the
real world usage when we have it.
The Configure integration section should more or less work the same as
before.
## Testing
- When utilising the onboarding flow for custom logs at
`/app/observabilityOnboarding/customLogs` can you:
- Create a custom integration? (It's worth verifying the network
requests, and the assets are installed).
- If you navigate forward, then back, make a change to the integration
fields, and navigate forward again is the previously created integration
deleted?
- Is the success callout with the integration name shown on the next
wizard panel?
- Do field validations work?
- Are errors displayed when you try to create an integration with a name
that already exists?
- Can you retry when there is a server error? (you can block network
requests to the custom integrations API to test this)
## Screenshots
## State machine diagram
(The top level management machine is super basic, so this is just the
create machine)
## Followups
- Tests (the current onboarding UI implementation doesn't have tests so
whilst it's not ideal technically this coverage stays the same)
- Storybook
- Replace other plugins' usage with xstate-utils (not urgent)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Yngrid Coello <yngrid.coello@elastic.co>
Addresses https://github.com/elastic/kibana/issues/160411
## Summary
This PR adds functionality for filtering out advanced settings that are
not relevant for serverless.
For context, we need to build an Advanced settings page in serverless
which only contains a set of the existing settings. We will reuse the
section registry (https://github.com/elastic/kibana/pull/163502) from
the original Advanced settings plugin as well as its UI components which
will also be extracted into a separate package. The app will be
registered from inside the `serverless` plugin.
In order to only display the settings that are relevant for serverless,
we need to make some changes to the uiSettings service. The
implementation in this PR leverages the existing `readonly` uiSettings
param and adds the `setAllowlist()` method which is called by the
serverless plugin to set an allowlist of setting keys.
**Testing in serverless:**
1. Set `advanced_settings.enabled: true` to enable the Advanced settings
app in serverless:
5b216c6ea9/config/serverless.yml (L53)
2. Start Es with `yarn es serverless --ssl` and Kibana with `yarn
serverless-{mode} --ssl` in any serverless mode.
3. Navigate to `app/management/kibana/settings`
4. Verify that the app only displays the settings from
`packages/serverless/settings/common/index.ts` (these are the settings,
relevant for all projects in serverless) as well as the settings from
the corresponding project package
`packages/serverless/settings/{mode}_project/index.ts`.
5. Verify that the app is functioning correctly.
**Testing in self-managed:**
1. Start Es with `yarn es snapshot` and Kibana with `yarn start`.
2. Go to Stack Management > Advanced settings
3. Verify that all settings are displayed as usual.
4. Verify that the app is functioning correctly.
If your team is a code owner of any of the serverless project plugins,
please review the corresponding package
`packages/serverless/settings/{search/observanility/security}_project/index.ts`
where you've been added as an owner and test in the serverless solution
accordingly.
<!---
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
-->
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
## Summary
This moves connectors functions to a shared package so Serverless Search
and Enterprise Search can both use it.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Current PR includes the next set of changes:
1. Moved `DiscoverGrid` component to a package `@kbn/unified-data-table`
and added `@elastic/kibana-data-discovery` as code owners.
2. Changed `@kbn/unified-data-table` package naming for data grid
related components and methods to correspond `UnifiedDataTable` instead
of `Discover`.
3. Moved hooks `useColumns` and `useRowHeightsOptions` to a package as
its logic belongs to `UnifiedDataTable`.
4. Renamed `DiscoverGridContext` to `UnifiedDataTableContext`.
5. Extended `UnifiedDataTable` interface and functionality with the next
customization options:
- `renderDocumentView?: (displayedRows:
DataTableRecord[],displayedColumns: string[]) => JSX.Element |
undefined;` - callback to render DocumentView when the document is
expanded
- `configRowHeight?: number;` - optional value for providing
configuration setting for UnifiedDataTable rows height
- `showMultiFields?: boolean;` - optional value for providing
configuration setting for enabling to display the complex fields in the
table. Default is true.
- `maxDocFieldsDisplayed?: number;` - optional value for providing
configuration setting for maximum number of document fields to display
in the table. Default is 50.
- `externalControlColumns?: EuiDataGridControlColumn[];` - optional
value for providing EuiDataGridControlColumn list of the additional
leading control columns. UnifiedDataTable includes two control columns:
Open Details and Select.
<img width="522" alt="Screenshot 2023-08-22 at 2 26 57 PM"
src="d796b9c8-2fef-4bcc-a3c9-9f5cc6349ab9">
- `externalAdditionalControls?: React.ReactNode;` - optional value for
providing the additional controls available in the UnifiedDataTable
toolbar to manage it's records or state. UnifiedDataTable includes
Columns, Sorting and Bulk Actions.
<img width="673" alt="Screenshot 2023-08-22 at 2 40 28 PM"
src="f7ac0c87-5310-49dd-9084-1ce01ca0f366">
- `rowsPerPageOptions?: number[];` - optional list of number type values
to set custom UnifiedDataTable paging options to display the records per
page.
- `renderCustomGridBody?: (args: EuiDataGridCustomBodyProps) =>
React.ReactNode;` - An optional function called to completely customize
and control the rendering of EuiDataGrid's body and cell placement.
<img width="1658" alt="Screenshot 2023-08-22 at 2 50 27 PM"
src="14adc18d-73af-40f5-9859-b3c708e265b1">
- `trailingControlColumns?: EuiDataGridControlColumn[];` - optional list
of the `EuiDataGridControlColumn` type for setting trailing control
columns standard for `EuiDataGrid`.
- `visibleCellActions?: number;` - optional value for a custom number of
the visible cell actions in the table
<img width="497" alt="Screenshot 2023-08-22 at 2 45 49 PM"
src="57ef3ad9-7401-46bb-9b38-cc8cca2e6a24">
- `externalCustomRenderers?: Record<string,(props:
EuiDataGridCellValueElementProps) => React.ReactNode>;` - an optional
settings for a specified fields rendering like links. Applied only for
the listed fields rendering:
<img width="1121" alt="Screenshot 2023-08-22 at 2 51 07 PM"
src="77501eae-3046-4a2c-90e1-2db487c21e2c">
- `consumer` - optional string value for the name of the
`UnifiedDataTable` consumer component or application.
6. Extended `UnifiedDataGrid` services with the two additional:
`storage: Storage;`
`data: DataPublicPluginStart; `
replaced `core: CoreStart;` with `theme: ThemeServiceStart;`, because
`core` is used only to get `theme`
7. Replaced `DocumentView` property with `renderDocumentView?:
(displayedRows: DataTableRecord[],displayedColumns: string[]) =>
JSX.Element | undefined;` callback function, which allows not to use
`DiscoverGridFlyout` definition for the documents rendering.
```
/**
* Document detail view component
*/
DocumentView?: typeof DiscoverGridFlyout;
```
8. Removed the next properties from the data table interface, because it
was used to render DiscoverGridFlyout:
```
/**
* Filters applied by saved search embeddable
*/
filters?: Filter[];
/**
* Query applied by KQL bar or text based editor
*/
query?: Query | AggregateQuery;
/**
* Saved search id used for links to single doc and surrounding docs in the flyout
*/
savedSearchId?: string;
```
9. Added usage examples and interface description to README file.
10. Changed grid styles naming from `.dscDiscoverGrid*` to
`.udtDataTable*`
11. Migrated discover plugin to use `UnifiedDataTable` instead of
`DiscoverGrid`
Extra changes were needed to avoid the circular dependancies:
- moved `DocViewFilterFn` and `FieldMapping` from discover plugin to
`packages/kbn-discover-utils/src/types.ts`
- added own `export type SortOrder = [string, string];` to avoid deps
for saved-search plugin
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This is part 1 of a series of PRs to expose the flamegraph to be used by
other plugins.
**The problem**
Currently for plugin-A to show data from plugin-B, it needs to add
dependency on plugin-B. If plugin-B wants to show data from plugin-A, it
also needs to add plugin-A as a dependency, and here is where the
problem happens. In such scenario, we have a cyclic dependency problem.
**The solution**
To solve this problem a new plugin is created, `profiling-data-access`.
This plugin exposes services that consumer plugins can call in other to
have the data they need to show on their end. The `profiling` plugin is
also using this new plugin. For now, only the flamegraph service is
available, The idea is to slowly migrate the data fetching from
profiling to this new plugin in other to facilitate the integration
across plugins.
**Why some many files?**
As I said, only the flamegraph logic was moved to the new plugin, but it
has many files that it needs to properly build the response of the
service call. I moved all these files to the `common` folder inside the
new plugin and adjusted the imports in the profiling plugin.
<img width="1032" alt="Screenshot 2023-08-31 at 09 29 27"
src="287bd28e-b834-45e0-8042-576d1fcff6cd">
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
### What this PR changes
branched from elastic/kibana/pull/163759
- Introduces new AppFeatures package `@kbn/security-solution-features`
with the common logic and `AppFeatureService` to apply offering specific
configurations for Security Solution features independently for
Serverless and ESS. This logic is replacing the earlier `AppFeatures` in
order to introduce new Kibana feature privileges for serverless PLIs so
that new Kibana privileges introduced for serverless PLIs do not
affect/show up as new Kibana feature privileges in ESS.
- Gates endpoint exceptions on alerts/rules based on serverless PLI
configurations. On serverless `Endpoint exceptions` should be
accessible/seen only on endpoint essentials/complete.
New AppFeatures logic architecture diagram:
**Note:** Corresponding API changes related to endpoint exceptions will
be in a new PR, along with the last set of UX changes for hiding the
`Endpoint exceptions` tab from the Rules details page.
### How to review
- Setup for _Servlerless_
- Run `yarn es snapshot` on a terminal window to start ES.
- Copy `config/serverless.security.yml` to
`config/serverless.security.dev.yml`
- Run `yarn serverless-security --no-base-path` on another terminal
window to start kibana in serverless mode
- Run `node
x-pack/plugins/security_solution/scripts/endpoint/endpoint_agent_emulator.js
--asSuperuser` on a new window and then select `1` to `Load Endoints`
and then `1` to `Run` the loader script. This will load some fake
agents/alerts data to test with.
### Tests (Serverless)
- with
`{ product_line: 'security', product_tier: 'essentials' }` or `{
product_line: 'security', product_tier: 'complete' }`
and
`{ product_line: 'endpoint', product_tier: 'essentials' }` or `{
product_line: 'endpoint', product_tier: 'complete' }`
1. Navigate to Rules>Shared exception lists via
`http://localhost:5601/app/security/exceptions`
2. Test that you can see `Endpoint Security Exception List` card on the
shared exception lists page.
3. Navigate to `Alerts` page via `app/security/alerts`, you should see
endpoint alerts. If not, then click on `Manage Rules` and then
disable/enable `Endpoint Security` rules. That should trigger alerts to
show up on the Alerts table.
4. Click on `View Details` button under `Actions` column. Once the
flyout is visible, click on `Take Action` and verify that `Add Endpoint
exception` is visible/enabled/clickable on the menu.
5. Click on `More actions` button under `Actions` column and verify that
`Add Endpoint exception` is visible/enabled/clickable on the menu.
6. Click on `Investigate in timeline` button under `Actions` column;
when the timeline view is visible and the alert item is displayed, click
on buttons mentioned in 4. and 5. above and verify the same.
7. Navigate to `Rules`>`DetectionRules`>`Endpoint Security` rule under
the `Rules` table. Select the `Alerts` tab.
8. Click and verify `View details`,`More actions` and `Investigate in
timeline` buttons same as in 4., 5., 6. above.
9. You should be able to see the `Endpoint exceptions` tab as well.
Click and verify that you can see the tab's content.
- with
`{ product_line: 'security', product_tier: 'essentials' }` or `{
product_line: 'security', product_tier: 'complete' }`
1. Edit `config/serverless.security.dev.yml` so that `endpoint` product
line item is commented out.
2. Test that you can not see `Endpoint Security Exception List` card on
the shared exception lists page.
3. Items 4. 5. 6. as above but the menu items should be disabled. This
can be verified with fake data only as with a real endpoint, endpoint
alerts are actually not visible at all.
### Tests (ESS)
On the ESS side, endpoint exceptions are not affected by this change and
work as usual based on index privileges.
---------
Co-authored-by: semd <sergi.massaneda@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: YulNaumenko <jo.naumenko@gmail.com>
Co-authored-by: Pablo Neves Machado <pablo.nevesmachado@elastic.co>
Co-authored-by: Pablo Machado <machadoum@gmail.com>
## [Security Solution] [Elastic AI Assistant] LangChain integration (experimental)
This PR integrates [LangChain](https://www.langchain.com/) with the [Elastic AI Assistant](https://www.elastic.co/blog/introducing-elastic-ai-assistant) as an experimental, alternative execution path.
### How it works
- There are virtually no client side changes to the assistant, apart from a new branch in `x-pack/packages/kbn-elastic-assistant/impl/assistant/api.tsx` that chooses a path based on the value of the `assistantLangChain` flag:
```typescript
const path = assistantLangChain
? `/internal/elastic_assistant/actions/connector/${apiConfig?.connectorId}/_execute`
: `/api/actions/connector/${apiConfig?.connectorId}/_execute`;
```
Execution of the LangChain chain happens server-side. The new route still executes the request via the `connectorId` in the route, but the connector won't execute the request exactly as it was sent by the client. Instead, the connector will execute one (or more) prompts that are generated by LangChain.
Requests routed to `/internal/elastic_assistant/actions/connector/${apiConfig?.connectorId}/_execute` will be processed by a new Kibana plugin located in:
```
x-pack/plugins/elastic_assistant
```
- Requests are processed in the `postActionsConnectorExecuteRoute` handler in `x-pack/plugins/elastic_assistant/server/routes/post_actions_connector_execute.ts`.
The `postActionsConnectorExecuteRoute` route handler:
1. Extracts the chat messages sent by the assistant
2. Converts the extracted messages to the format expected by LangChain
3. Passes the converted messages to `executeCustomLlmChain`
- The `executeCustomLlmChain` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/execute_custom_llm_chain/index.ts`:
1. Splits the messages into `pastMessages` and `latestMessage`, where the latter contains only the last message sent by the user
2. Wraps the conversation history in the `BufferMemory` LangChain abstraction
3. Executes the chain, kicking it off with `latestMessage`
```typescript
const llm = new ActionsClientLlm({ actions, connectorId, request });
const pastMessages = langchainMessages.slice(0, -1); // all but the last message
const latestMessage = langchainMessages.slice(-1); // the last message
const memory = new BufferMemory({
chatHistory: new ChatMessageHistory(pastMessages),
});
const chain = new ConversationChain({ llm, memory });
await chain.call({ input: latestMessage[0].content }); // kick off the chain with the last message
};
```
- When LangChain executes the chain, it will invoke `ActionsClientLlm`'s `_call` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/llm/actions_client_llm.ts` one or more times.
The `_call` function's signature is defined by LangChain:
```
async _call(prompt: string): Promise<string>
```
- The contents of `prompt` are completely determined by LangChain.
- The string returned by the promise is the "answer" from the LLM
The `ActionsClientLlm` extends LangChain's LLM interface:
```typescript
export class ActionsClientLlm extends LLM
```
This let's us do additional "work" in the `_call` function:
1. Create a new assistant message using the contents of the `prompt` (`string`) argument to `_call`
2. Create a request body in the format expected by the connector
3. Create an actions client from the authenticated request context
4. Execute the actions client with the request body
5. Save the raw response from the connector, because that's what the assistant expects
6. Return the result as a plain string, as per the contact of `_call`
## Desk testing
This experimental LangChain integration may NOT be enabled via a feature flag (yet).
Set
```typescript
assistantLangChain={true}
```
in `x-pack/plugins/security_solution/public/app/app.tsx` to enable this experimental feature in development environments.
UX: https://github.com/elastic/security-team/issues/7310
## Summary
* It creates an Upselling package to share the service and components
between ESS and Serverless plugins
* It implements upselling for entity analytics on serverless by
replicating the ESS approach
ESS
<img width="1520" alt="Screenshot 2023-08-17 at 13 34 59"
src="95c2c94e-7ab3-4d9f-aa24-b3e9c00eb3ed">
Serverless
<img width="1523" alt="Screenshot 2023-08-17 at 13 39 25"
src="618ce9dc-ef4e-469d-884a-dfb09834d0b0">
We are not displaying the upgrade button because we still don't know how
to link to the cloud settings page.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Closes https://github.com/elastic/kibana/issues/161906
Related to:
https://github.com/elastic/observability-dev/discussions/2787
(_internal_)
This add a new plugin `apm_data_access` that contains the APM query
targets (indices to query for APM data).
This plugin can be consumed by apm and any other plugin, making it
possible for other plugins to know about the configured APM query
targets.
## Example:
APM query targets can be specified in kibana[.dev].yml using
`xpack.apm.indices.{dataset}: some-index-*` for instances:
```yml
xpack.apm.indices.transaction: apm-*
```
See all config options on:
https://www.elastic.co/guide/en/kibana/current/apm-settings-kb.html#general-apm-settings-kb
Query targets can also be specified via the UI (and persisted in a saved
object) via the settings page: `/app/apm/settings/apm-indices`
**Retrieving the query targets**
Query targets can be retrieved from other plugins via `getApmIndices`:
```ts
const apmIndices = await plugins.apmDataAccess.setup.getApmIndices(savedObjects.client);
```
TODO:
- [x] Add SO client and fetch space aware index config (see
4d7f626da0/x-pack/plugins/apm/server/routes/settings/apm_indices/get_apm_indices.ts (L32-L44))
- [ ] Add simple APM client for querying apm data
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
While working to extract various portions of the `advancedSettings`
plugin into packages, I found the `ComponentRegistry` in the plugin to
have a number of issues that contributed to a fairly bad UX:
- the API allows for adding/overriding the title, subtitle and footer of
the Advanced Settings page, but only the footer is rendered.
- the API is available to all plugins, but only renders a single
entry... so depending on the plugin load order, the render is not
guaranteed.
- filtering the footer in or out of the display is delegated to the
component itself, so:
- it only takes effect on render.
- the count is only updated if you click on the page that contains it,
but that logic is currently broken.
- the error message is inaccurate.
This PR fixes those issues and more:
- extracts the registry into its own package.
- changes the API to allow for multiple sections from multiple plugins.
- changes the API to filter these sections from the plugin, rather than
from each individual component.
- fixes state management to show sections, keep counts accurate, etc.
---------
Co-authored-by: Vadim Kibana <82822460+vadimkibana@users.noreply.github.com>
closes [#163491](https://github.com/elastic/kibana/issues/163491)
## Summary
This PR creates a new package that contains a utility API that helps to
generate the JSON with the attributes required to render a Lens chart
with the `EmbeddableComponent`.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This closes https://github.com/elastic/kibana/issues/161960, a basic
integration will now be created whilst onboarding logs (though the
custom logs flow).
This implements the *initial* version of this work, and does not include
things like adding a dataset to an existing integration.
## UI / UX
General:
Naming conflict errors:
Lack of permissions error:
General errors:
Success callout on the next panel:
Delete previous flow (happens in the background):
## Pointers for reviewers / next steps
- This PR also creates a new package for the `useTrackedPromise` hook,
as this is used in several places and I didn't want to just duplicate it
again (I haven't replaced other current uses in this PR, but will as a
followup).
- `useFetcher` was avoided as A) it's very tightly coupled with the
observability onboarding server route repository (and `callApi` is
scoped to this) and I wanted to call an "external" API in Fleet and B) I
wanted explicit control over when the request is dispatched (not on
mount), and whilst this can sort of be achieved by not returning a
promise from the callback it gets quite messy. I also wanted more
granular error handling control.
- Moving forward I think we'll need to enhance the state management of
the plugin. We'll want to add the ability to "add to existing
integration" and this is going to make the state more complex (even with
chunks of this functionality likely moved to it's own package). I did
actually have the Wizard state moved in to a constate container at one
point (as a starter) but I reverted this commit to make the changeset
less intrusive. It's for this same reason that, for now, I haven't
focussed too closely on extracting things like generating the friendly
error messages etc as we'll likely want to extract some of the "create
integration" hooks / UI in to a standalone package so they can be used
elsewhere (not just onboarding). There are also quite a few `
eslint-disable-next-line react-hooks/exhaustive-deps` rules in the
plugin at the moment due to the references not being stable, we could
improve that at the same time as any state changes.
- You can technically navigate directly to
`/fox/app/observabilityOnboarding/customLogs/installElasticAgent`, but
no state is stored in the URL, so nothing is rehydrated resulting in a
very empty configuration. I'm not entirely sure this is a behaviour we
want, but for now I've just made the callout conditional on state
existing (so coming from the previous panel).
- The Fleet custom integrations API now throws a 409 (conflict) when
using a name that already exists.
## Testing
- Head to `/app/observabilityOnboarding` to trigger the onboarding flow
- Select "Stream log files"
- When hitting "continue" an integration should be created in the
background (check the network requests for
`api/fleet/epm/custom_integrations`)
- When continuing (to install shipper), then going back **and** making
changes to your integration options, when clicking continue again there
should be a network request that deletes the previously created
integration (to clean things up). This should be seamless to the user.
- You should not be able to use a name that already exists (for an
existing custom integration)
- General errors (like permission issues, asset installation issues)
should display at the bottom
- When you hit the next panel (install shipper) there should be a
success callout that also contains the name of the integration that was
created
## In progress
~Two changes still in progress, but they don't need to hold up the
review (8.10 coming soon 👀):~
- ~To have a friendlier error for permissions issues (not just
"forbidden")~
- ~Fleet API integration test for the naming collision~
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
- Closes https://github.com/elastic/kibana/issues/155216
## Summary
This PR replaces shard falures toasts with inline warnings in Discover.
- [x] Intercept shard failures in Discover main app
- [x] Show inline warnings above the grid instead
- [x] Handle NoResultsFound case too
- [x] Implement for Discover context app
- [x] Implement for saved search embeddable on Dashboard
- [x] Can we inline timeouts too?
- [x] Check SQL view
- [x] Add tests
Discover view with shard failures
<img width="400" alt="Screenshot 2023-07-06 at 14 23 48"
src="a0799aa0-9d2e-42ee-b89b-e0e1180220a5">
Discover view with shard failures (and no results)
<img width="400" alt="Screenshot 2023-07-07 at 13 24 50"
src="28dc2bad-9776-4aa9-8f51-219d1c87487a">
Dashboard view with shard failures
<img width="400" alt="Screenshot 2023-07-06 at 16 15 49"
src="2a68c19a-1ca8-4f10-a9f5-4aa56f9160b0">
Surrounding documents view with shard failures
<img width="400" alt="Screenshot 2023-07-10 at 17 26 31"
src="ade63cfe-a1c2-4c22-8823-58dcfef9357f">
Discover view with timeouts
<img width="400" alt="Screenshot 2023-07-07 at 16 47 27"
src="0101be19-a555-4f96-b963-7fe418d51fb5">
Dashboard view with timeouts
<img width="400" alt="Screenshot 2023-07-07 at 16 48 18"
src="82979365-4129-4385-8a13-9c139e1acbf1">
Surrounding documents view with timeouts
<img width="400" alt="Screenshot 2023-07-11 at 15 03 41"
src="0ea41b79-ac6b-4456-9cfa-0d038b10da7d">
## Testing
For testing please uncomment
3f102cf688/src/plugins/data/common/search/search_source/search_source.ts (L922)
or
3f102cf688/src/plugins/data/common/search/search_source/search_source.ts (L547)
and switch to `kibana*` data view.
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [x] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
---------
Co-authored-by: Davis McPhee <davismcphee@hotmail.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/162293 and partially
addresses [154601](https://github.com/elastic/kibana/issues/154601)
Removes the screenshot diagnostic tool but keeps the browser check
Removes PngV1ExportType from core and the export types registry
### Before
There were two steps so the EuiSteps component definitely made more
sense.
### After
I removed the use of the EuiSteps component since there's only the
browser check. Since the EuiSteps also showed some validation, I added a
callout to let users know the status of the diagnostic.
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
> Pre-req for https://github.com/elastic/kibana/issues/56406
## Summary
We've had a long-standing problem in Kibana around our use of React
context, particularly with EUI and i18n. There hasn't existed an
idempotent context structure, and that has lead to a lot of unexpected
results, (e.g. missing translations, inconsistent dark mode, excess
context providers, etc).
The biggest change coming from this PR is knowing exactly which provider
to use in a particular use case. This means, for example,
`ReactDOM.render` calls won't be missing `i18n` or `theme` due to a
missing context. It also allows consumers to use `darkMode` without
having to read the `uiSetting` themselves, instead allowing the context
to do it for them.
We also haven't been honoring the intended [`EuiProvider`
API](https://eui.elastic.co/#/utilities/provider#theming-and-global-styles)...
in some cases we've been creating and re-creating the Emotion caches,
often by copy/paste of the cache code. We've also been nesting
`EuiThemeProvider` contexts unnecessarily-- thinking we need to render a
theme provider in an isolated component-- which renders an additional
`span` element into the DOM.
This PR attempts to address this inconsistency by creating a set of
context providers divided by use case:
### `KibanaRootContextProvider`
A root context provider for Kibana. This is the top level context
provider that wraps the entire application. It is responsible for
initializing all of the other contexts and providing them to the
application. It's provided as a package for specific use cases, (e.g.
the `RenderingService`, cases where we replace the entire page content,
Storybook, testing, etc), but not intended for plugins.
### `KibanaRenderContextProvider`
A render context provider for Kibana. This context is designed to be
used with ad-hoc renders of React components, (usually with
`ReactDOM.render`).
### `KibanaThemeContextProvider`
A theme context provider for Kibana. A corollary to EUI's
`EuiThemeProvider`, it uses Kibana services to ensure the EUI Theme is
customized correctly.
### (deprecated) `KibanaStyledComponentsThemeProvider`
A styled components theme provider for Kibana. This package is supplied
for compatibility with legacy code, but should not be used in new code.
## Deprecation strategy
This PR does *not* change any use of context by consumers. It maps the
existing contexts in `kibanaReact` to the new contexts, (along with the
loose API). This means that we won't have completely fixed all of our
dark mode issues yet. But this is necessary to keep this PR focused on
the change, rather than drawing in a lot of teams to review individual
uses.
We should, however, see an immediate performance improvement in the UI
from the reduction in `EuiProvider` calls.
## Open questions
- [ ] Does it make sense to expose a `useTheme` hook from
`@kbn/react-kibana-context-theme` to replace `useEuiTheme`?
## Next steps
- [ ] Update deprecated uses to new contexts.
- [ ] Audit and update calls to `ReactDOM.render`.
- [ ] Add ESLint rule to warn for use of EUI contexts.
- [ ] Delete code from `kibanaReact`.
This PR makes the following changes:
- Update look & feel of contextual insights (previously called prompts)
according to the new design that is being developed. Some things might
still change, but hopefully not too much.
- Move all the Observability AI Assistant (previously called CoPilot)
code into a separate plugin for better isolation, more specific code
ownership and to solve some circular dependency issues
- Use connectors instead of a kibana.yml setting
Note: for OpenAI, the model is currently hardcoded to `gpt-4` until
https://github.com/elastic/kibana/issues/162204 has been addressed.
557676b6-065a-4b6f-86b2-1f0c2fd5e07e
---------
Co-authored-by: Coen Warmer <coen.warmer@gmail.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
- Originally Kibana's `http` service did not support receiving streams,
that's why we used plain `fetch` for this. This has been fixed in
#158678, so this PR updates the streaming helpers to use Kibana's `http`
service from now on.
- The PR also breaks out the response stream code into its own package
and restructures it to separate client and server side code. This brings
down the `aiops` bundle size by `~300KB`! 🥳
- The approach to client side throttling/buffering was also revamped:
There was an issue doing the throttling inside the generator function,
it always waited for the timeout. The buffering is now removed from
`fetchStream`, instead `useThrottle` from `react-use` is used on the
reduced `data` in `useFetchStream`. Loading log rate analysis results
got a lot snappier with this update!
## Summary
Implementation of serverless-specific pages within the Unified IA
Navigation.
#### Links implemented:
- `Machine Learning`
- Landing page created on serverless only
- All links in the landing page go to `/ml` app
- `Dev Tools`
- Links directly to `/dev_tools` app
#### Links not implemented:
```// TODO: in a follow-up PR```
- Project Settings
- Change the _Settings_ name by _Project Settings_
- Modify the landing page items according to the design
## Changes
### Plugin contract changes
The Machine Learning landing page is the first page that is only available on serverless and should not exist in ess (there are more of this kind in the pipeline), so this PR implements the foundations to enable the _security_solution_serverless_ plugin to implement its own page components, configure the link definition and create new routes to render them in the Security Solution application.
These new APIs can be called from either `security_solution_serverless` or `security_solution_ess`, allowing those plugins to have their own offering-specific pages.
The new APIs exposed in the security_solution public contract are the following:
- `extraAppLinks$`: Observable to add extra app_links into the application links configuration, so they are stored and included in the SecuritySolution plugin `deepLinks` registry, to make them accessible from anywhere in the application using the `chrome.navLinks` API.
- `extraRoutes$`: Observable to add extra routes into the main Router, so it can render the new page components. These additional routes are appended after the "sub-plugin" (_alerts_, _timeline_, ...) routes, so it is not possible to override an existing route path.
### New `security-solution-navigation` package
Since now we need to use the same navigation components and hooks in different plugins, these functionalities have been extracted to the `@kbn/security-solution-navigation` package, which all Security plugins will depend on (generic, serverless, and ess).
The modules exposed by this package have been extracted from the main security_solution plugin and standardized. They include the Landing pages components (new [storybook](https://ci-artifacts.kibana.dev/storybooks/pr-161667/394abe76676c6a76b2982c1d3f5bb675739c3477/security_solution_packages/index.html?path=/story/landing-links-landing-links-icons-categories--landing-links-icons-categories) available), navigation hooks, and link utilities. Also, some types and constants have been moved to this package.
A new context provider has also been created, which needs to be in place in order to use this package. The `<NavigationProvider core={core}>` is required for the package functionalities to have access to the Kibana core navigation APIs: `navigateToUrl`, `navigateToApp`, and `getUrlForApp`.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: YulNaumenko <jo.naumenko@gmail.com>
