The EBT pacakge has been moved to a separate NPM package
([@elastic/ebt](https://www.npmjs.com/package/@elastic/ebt))
The npm package is on version `0.0.x` until we finish the reviews then
i'll publish the `1.0.0` version before merging this PR.
The PR is mostly code deletes after moving the code to the public ebt
github repo https://github.com/elastic/ebt
The significant changes are:
1. removed the `packages/analytics/ebt` package from kibana
2. remove @kbn/ebt references in favor of the npm package.
3. Added a util package to provide the package with the telemetry
endpoint and headers
This was previously backed into the package but now i've rewired it be
provided from Kibana, this way we have more control over the URL and
headers we use to send EBT telemetry for our elastic endpoint, which
will probably be different between users of this package and this way
we'll also avoid republishing the package if we ever want to change
these details.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Alejandro Fernández Haro <afharo@gmail.com>
## Summary
At the moment, our package generator creates all packages with the type
`shared-common`. This means that we cannot enforce boundaries between
server-side-only code and the browser, and vice-versa.
- [x] I started fixing `packages/core/*`
- [x] It took me to fixing `src/core/` type to be identified by the
`plugin` pattern (`public` and `server` directories) vs. a package
(either common, or single-scoped)
- [x] Unsurprisingly, this extended to packages importing core packages
hitting the boundaries eslint rules. And other packages importing the
latter.
- [x] Also a bunch of `common` logic that shouldn't be so _common_ 🙃
### For maintainers
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Related to #183220
## Summary
This PR extracts `getEcsGroups` to a package to save ECS groups in the
Alert As Data (AAD) document for the metric threshold rule.
### 🧪 How to test
- Create a metric threshold rule with multiple groups (both ECS and
non-ECS fields)
- Check the related AAD document; you should be able to see the ECS
fields at the root level and not see non-ECS fields there
- Check the same information for the recovered alerts
- Rules without group by should work as before
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Contributes to:
- https://github.com/elastic/security-team/issues/9954
The PR contains the base for the `DistributionBar` component to be used
in the new Entity Flyout Insights.
Not included:
- badges per distribution with the number of documents and pretty names
- on hover interaction
## Screenshots
<img width="980" alt="Screenshot 2024-07-17 at 15 13 48"
src="https://github.com/user-attachments/assets/f2ca53ee-c054-4923-aa3f-7dd4017754cb">
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
- [x] This is an improvement pr to move all the avc banner logic into a
reusable kibana package (security solution and fleet integrations)
- [x] Compresses the svg used in the banner's background
- [x] Fixes a bug where the blog link didn't previously open in a new
tab
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Close https://github.com/elastic/kibana/issues/183686
Adds a new sorting option "Recently viewed". Recently viewed dashboards
will be shown at the top of the list. The remaining dashboards are
sorted by last updated at.
- This is a default option unless another option was explicitly selected
before (and saved in local storage)
- If there are no recently viewed dashboards, then this sorting option
is hidden, and "last updated at" is default like before
- This option is only added to the dashboard listing
Implementation:
- Recently viewed dashboard are stored in local storage as a queue with
20 items max
- I reused the existing RecentlyAccessedService we've been using for
sidenav's recently viewed section. For this, I moved it to a separate
package. The service already handles a lot of edge cases like spaces,
key hashing, and deduping.
- The sorting part in EUITable is a bit hacky. It doesn't support custom
internal sorting (like we do with title and lastUpdatedAt), so I had to
sort the list myself and then tell EUITable not to do any sorting in
case "Recently viewed" option is selected. [slack
discussion](https://elastic.slack.com/archives/C7QC1JV6F/p1720008717120589)
<img width="1265" alt="Screenshot 2024-07-05 at 10 59 25"
src="9cc46fd2-4270-494f-9272-302007a7efc0">
## Summary
Renames `@kbn/text-based-languages` plugin to `@kbn/esql` plugin. This
has been discussed internally, the rationale is that now there will be
only one language: ES|QL; and we may use this plugin for ES|QL related
HTTP routes.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Adds solution-agnostic components to create hierarchical alerts grouping
UIs, adapting the original implementation from Security Solution.
Closes#184398
## To Verify
For existing usages of the `@kbn/grouping` package: verify that the
grouped UIs work correctly (Security Alerts, Cloud Security Posture).
New alerting UI components: checkout
https://github.com/elastic/kibana/pull/183114 (PoC PR), where the
updated `@kbn/grouping` package and these new components are used in
Observability's main Alerts page.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Gerard Soldevila <gerard.soldevila@elastic.co>
Co-authored-by: Vadim Kibana <82822460+vadimkibana@users.noreply.github.com>
Co-authored-by: Alex Szabo <alex.szabo@elastic.co>
Co-authored-by: Tre <wayne.seymour@elastic.co>
## Summary
Renames the experimental asset_manager plugin (never
documented/officially released) into entity_manager. I've used `node
scripts/lint_ts_projects --fix` and `node scripts/lint_packages.js
--fix` to help with the procedure and also renamed manually the
asset_manager references left.
The change also removes the deprecated asset_manager code, including the
`assetManager.alphaEnabled` plugin configuration. This means
entityManager plugin will be enabled by default.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This consumes the security solution's API key flyout and token selector
in Serverless search to have a consistent experience across the stack.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR updates the Rollup badge in the indices list and the data views
list so that it provides deprecation information. The tooltip component
is stored in a new package to avoid cyclical dependencies as it is used
in both the Index management plugin and in the Data views management
plugin.
<details><summary>Screenshots</summary>
<img width="1679" alt="Screenshot 2024-06-17 at 18 08 35"
src="1d3daa9d-3f62-49f8-803f-1b3d4605f9a4">
<img width="1679" alt="Screenshot 2024-06-17 at 18 08 53"
src="2e88d119-88e7-4d01-bab9-bc2add82b91c">
<img width="1679" alt="Screenshot 2024-06-17 at 18 09 18"
src="29d3d3f2-de46-45ea-96a1-b83495d122df">
</details>
**How to test:**
1. Start Es and Kibana
2. Add the sample data "Sample web logs"
3. Elasticsearch only allows creating a rollup job if there is an
existing rollup usage in the cluster. To simulate rollup usage, create a
mock rollup index through Console:
```
PUT /mock_rollup_index
{
"mappings": {
"_meta": {
"_rollup": {
"id": "logs_job"
}
}
}
}
```
4. Create a sample rollup job through Console:
```
PUT _rollup/job/logs_job
{
"id": "logs_job",
"index_pattern": "kibana_sample_data_logs",
"rollup_index": "rollup_logstash",
"cron": "* * * * * ?",
"page_size": 1000,
"groups": {
"date_histogram": {
"interval": "60m",
"delay": "7d",
"time_zone": "UTC",
"field": "@timestamp"
},
"terms": {
"fields": [
"geo.src",
"machine.os.keyword"
]
},
"histogram": {
"interval": "1003",
"fields": [
"bytes",
"memory"
]
}
}
}
```
5. Delete the mock rollup index since it causes issues for the rollup
API that we use to fetch rollup indices:
`DELETE /mock_rollup_index`
6. Navigate to Index Management and toggle the "Include rollup indices"
switch
7. Verify that the rollup indices have the deprecation badge.
8. Navigate to Data Views and create a rollup data view with index
pattern that matches the created rollup index (`rollup*`).
9. In the list of data views, verify that the rollup data view has the
correct rollup deprecation badge.
10. Click on the rollup data view and verify that the details panel also
has the deprecation badge.
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [x] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [x] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [x] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/66716
Improves code editors in Anomaly detection, Data frame analytics and
Transform wizards with autocomplete, data types validation and inline
documentation from elasticsearch specification.
Adds a package with JSON schemas extracted from the [openapi output of
elasticsearch-specification](https://github.com/elastic/elasticsearch-specification/tree/main/output/openapi).
Schema file is generated per editor/endpoint, keeping only relevant
components.
To test the script locally, execute
```
yarn run jsonSchema
```
from `/x-pack/packages/ml/json_schemas`.
By default it assumes that your `elasticsearch-specification` folder is
located next to the `kibana` repo, but you can also provide a path to
`openapi` file as a parameter, e.g. `yarn run jsonSchema
/Users/my_user/dev/elasticsearch-specification/output/openapi/elasticsearch-serverless-openapi.json`
#### How JSON files are served
JSON files are asynchronously imported at the React component level and
passed to the `CodeEditor` instances on mount.
Initially I tried different approaches to take advantage of CDN, but
unfortunately it didn't work out:
##### Using static assets
I made an attempt to retrieve a URL to the JSON schema file as a static
asset using Kibana service
```typescript
const schemaJsonAsset = http?.staticAssets.getPluginAssetHref('my_schema.json') ?? '';
```
and passing it as part of the schema definition, but the browser was
blocking a request 🤔
##### Using raw loader
Approach with a raw loader also didn't succeed.
```typescript
import mySchema from '!!raw-loader!./my_schema.json';
```
<details>
<summary>See error </summary>
```
ERROR in
./public/app/sections/create_transform/components/advanced_pivot_editor/my_schema.json
(/Users/dimaarnautov/Repos/kibana/node_modules/raw-loader/dist/cjs.js!./public/app/sections/create_transform/components/advanced_pivot_editor/my_schema.json)
│ Module parse failed: Unexpected token 'e', "export def"... is not
valid JSON while parsing 'export default "{\n \"type\": \"object\'
│ File was processed with these loaders:
│ * ../../../node_modules/raw-loader/dist/cjs.js
│ You may need an additional loader to handle the result of these
loaders.
│ SyntaxError: Unexpected token 'e', "export def"... is not valid JSON
while parsing 'export default "{\n \"type\": \"object\'
│ at JSON.parse (<anonymous>)
│ at parseJson
(/Users/dimaarnautov/Repos/kibana/node_modules/json-parse-better-errors/index.js:7:17)
│ at JsonParser.parse
(/Users/dimaarnautov/Repos/kibana/node_modules/webpack/lib/JsonParser.js:16:16)
│ at
/Users/dimaarnautov/Repos/kibana/node_modules/webpack/lib/NormalModule.js:482:32
│ at
/Users/dimaarnautov/Repos/kibana/node_modules/webpack/lib/NormalModule.js:358:12
│ at
/Users/dimaarnautov/Repos/kibana/node_modules/loader-runner/lib/LoaderRunner.js:373:3
│ at iterateNormalLoaders
(/Users/dimaarnautov/Repos/kibana/node_modules/loader-runner/lib/LoaderRunner.js:214:10)
│ at iterateNormalLoaders
(/Users/dimaarnautov/Repos/kibana/node_modules/loader-runner/lib/LoaderRunner.js:221:10)
│ at
/Users/dimaarnautov/Repos/kibana/node_modules/loader-runner/lib/LoaderRunner.js:236:3
│ at runSyncOrAsync
(/Users/dimaarnautov/Repos/kibana/node_modules/loader-runner/lib/LoaderRunner.js:130:11)
│ at iterateNormalLoaders
(/Users/dimaarnautov/Repos/kibana/node_modules/loader-runner/lib/LoaderRunner.js:232:2)
│ at
/Users/dimaarnautov/Repos/kibana/node_modules/loader-runner/lib/LoaderRunner.js:205:4
│ at
/Users/dimaarnautov/Repos/kibana/node_modules/webpack/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:85:15
│ at processTicksAndRejections (node:internal/process/task_queues:77:11)
```
</details>
### Checklist
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
Introducing the `search_homepage` plugin along with integration into
`enterprise_search` and `serverless_search` behind a feature flag. This
will allow implementing the feature gated behind the feature flag.
To test these changes you can enable the feature flag with the Kibana
Dev Console using the following command:
```
POST kbn:/internal/kibana/settings/searchHomepage:homepageEnabled
{"value": true}
```
You can then disable the feature flag with the following command:
```
DELETE kbn:/internal/kibana/settings/searchHomepage:homepageEnabled
```
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Close https://github.com/elastic/kibana-team/issues/898
- Show createdAt, createdBy, updatedAt, updatedBy in info flyout. Add a
bit of special handling for managed objects and when info is not
available.
- I had to extract some components into a separate package to use them
in contentEditor package
- tiny tweaks to column width and "no creator" state
**Unknown creator:**
**For managed objects:**
**Just created, no updates yet**
## Summary
This is a PR to add a new backend plugin (frontend will be done in
separate [PR](https://github.com/elastic/kibana/pull/184546)).
The purpose of the plugin is to provide a set of API routes that is used
to perform a variety of GenAI workflows to generate new integrations
based on provided inputs.
It reuses the existing GenAI connectors for its LLM communication, and
provides a set of API's to create ECS mapping, Categorization, Related
Fields and an API to generate the actual integration package zip, which
is forwarded to the UI component.
### Planned follow-up changes:
As the PR is getting way too large, some planned changes would be added
in much smaller follow-ups. This includes mostly more improved try/catch
for certain routes, adding debug/error log entries where relevant,
especially for the API endpoints themself, some more unit and end2end
tests.
- OpenAPI spec for the API will be handled in a separate PR
- All the missing unit tests will be added as a followup PR
### Testing
The `integration_assistant` plugin will be disabled by default while
it's being implemented so we can iterate and merge partial PRs without
interfering with the releases. This config will work as our feature
flag:
6aefd4ff7b/x-pack/plugins/integration_assistant/server/config.ts (L11-L13)
To test it add this to your _kibana.dev.yml_:
```
xpack.integration_assistant.enabled: true
```
### Checklist
Delete any items that are not applicable to this PR.
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: Patryk Kopycinski <contact@patrykkopycinski.com>
Co-authored-by: Sergi Massaneda <sergi.massaneda@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Bharat Pasupula <saibharatchandra.pasupula@elastic.co>
Co-authored-by: Bharat Pasupula <123897612+bhapas@users.noreply.github.com>
resolves: https://github.com/elastic/kibana/issues/181325
## Summary
Adds a new task claiming strategy `unsafe_mget`, which can be used instead of
the default one `default`. Add the following to your `kibana.yml` to
enable it:
xpack.task_manager.claim_strategy: 'unsafe_mget'
## Description
In this PR, we implemented a view for managing inference endpoints. The
changes include the following items for both **Serverless** and
**Stack**.
- A blank page will be displayed if no inference endpoints are
available.
- A page displaying a list of inference endpoints. The user can view
various details about each endpoint, such as the endpoint itself, the
provider, and the type. The table supports pagination and sorting.
- Users can add a new inference endpoint using Elasticsearch models and
third-party APIs, including Hugging Face, Cohere, and OpenAI.
To keep the changes in this PR manageable, the following items are **out
of scope** but will be added in subsequent PRs
- Option to delete an inference endpoint
- Filtering and Search bar
- Information about allocations, thread.
- Icons for **Provider**
- Deployment status of underlying trained models
## Empty page in Stack Management
e2064ee8-3623-457f-8a04-19603e97e815
## Page with all inference endpoints in Stack Management
89bec450-1569-4425-b013-5058b577b95a
## Inference Endpoints Management in Serverless
bd8b6b71-0e09-49f4-aa9a-19338a1da225
---------
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR creates a package to expose tools for managing feature flags in
the ResponseOps codebase.
Usage:
```
const featureFlagService = createFeatureFlagService(['test.myFeature', 'test.myFeature.subFeature']);
if (featureFlagService.isFeatureFlagSet('test.myFeature')) {
// my feature code
}
```
The code is typed so if you do
`featureFlagService.isFeatureFlagSet('foo')` and `foo` is not part of
the feature flags set TS will report an error.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### For maintainers
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Create the Investigate plugin (naming TBD). Part of
https://github.com/elastic/kibana/pull/183293, splitting up the work in
several PRs.
The investigate plugin is mostly a registry to allow plugins to register
their widgets without creating dependency issues.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## 📓 Summary
Closes https://github.com/elastic/observability-dev/issues/3331
Given the needs described in the linked issue about having a centralized
and async way to consume field metadata across Kibana, this work focuses
on providing server/client services to consume field metadata on demand
from static ECS definition and integration manifests, with the chance to
extend further the possible resolution sources.
3b2d9027-5c29-4081-ab17-1b43618c62a7
## 💡 Reviewers hints
This PR got quite long as it involves and touches different parts of the
codebase, so I'll break down the interesting parts for an easier review.
More details, code examples and mechanics description can be found in
the README file for the plugin.
### `@kbn/fields-metadata-plugin`
To avoid bundling and consuming the whole ECS static definition
client-side, a new plugin `@kbn/fields-metadata-plugin` is created to
expose the server/client services which enable retrieving only the
fields needed on a use-case basis.
### FieldsMetadataService server side
A `FieldsMetadataService` is instantiated on the plugin setup/start
server lifecycle, exposing a client to consume the fields and setup
tools for registering external dependencies.
The start contract exposes a `FieldsMetadataClient` instance. With this,
any application in Kibana can query for some fields using the available
methods, currently:
- `FieldsMetadataClient.prototype.getByName()`: retrieves a single
`FieldMetadata` instance.
- `FieldsMetadataClient.prototype.find()`: retrieves a record of
matching `FieldMetadata` instances.
`FieldsMetadataClient` is instantiated with the source repositories
dependencies. They act as encapsulated sources which are responsible for
fetching fields from their related source. Currently, there are 2 field
repository sources used in the resolution step, but we can use this
concept to extend the resolution step in future with more sources (LLM,
OTel, ...).
The currently used sources are:
- `EcsFieldsRepository`: allows fetching static ECS field metadata.
- `IntegrationFieldsRepository`: allows fetching fields from an
integration package from EPR, where the fields metadata are stored. To
correctly consume these fields, the `fleet` plugin must be enabled,
otherwise, the service won't be able to access the registered fields
extractor implemented with the fleet services.
As this service performs a more expensive retrieval process than the
`EcsFieldsRepository` constant complexity access, a caching layer is
applied to the retrieved results from the external source to minimize
latency.
### Fields metadata API
To expose this service to the client, a first API endpoint is created to
find field metadata and filter the results to minimize the served
payload.
- `GET /internal/fields_metadata/find` supports some initial query
parameters to narrow the fields' search.
### FieldsMetadataService client side
As we have a server-side `FieldsMetadataService`, we need a client
counterpart to consume the exposed API safely and go through the
validation steps.
The client `FieldsMetadataService` works similarly to the server-side
one, exposing a client which is returned by the public start contract of
the plugin, allowing any other to directly use fields metadata
client-side.
This client would work well with existing state management solutions, as
it's not decoupled from any library.
### useFieldsMetadata
For simpler use cases where we need a quick and easy way to consume
fields metadata client-side, the plugin start contract also exposes a
`useFieldsMetadata` react custom hook, which is pre-created accessing
the FieldsMetadataService client described above. It is important to
retrieve the hook from the start contract of this plugin, as it already
gets all the required dependencies injected minimizing the effort on the
consumer side.
The `UnifiedDocViewer` plugin changes exemplify how we can use this hook
to access and filter fields' metadata quickly.
### `registerIntegrationFieldsExtractor` (@elastic/fleet)
Getting fields from an integration dataset is more complex than
accessing a static dictionary of ECS fields, and to achieve that we need
access to the PackageService implemented by the fleet team.
To get access to the package, maintain a proper separation of concerns
and avoid a direct dependency on the fleet plugin, some actions were
taken:
- the `PackageService.prototype.getPackageFieldsMetadata()` method is
implemented to keep the knowledge about retrieving package details on
this service instead of mixing it on parallel services.
- a fleet `registerIntegrationFieldsExtractor` service is created and
used during the fleet plugin setup to register a callback that accesses
the service as an internal user and retrieves the fields by the given
parameters.
- the fields metadata plugin returns a
`registerIntegrationFieldsExtractor` function from its server setup so
that we can use it to register the above-mentioned callback that
retrieves fields from an integration.
This inverts the dependency between `fields_metadata` and `fleet`
plugins so that the `fields_metadata` plugin keeps zero dependencies on
external apps.
## Adoption
We currently have places where the `@elastic/ecs` package is directly
accessed and where we might be able to refactor the codebase to consume
this service.
**[EcsFlat usages in
Kibana](https://github.com/search?q=repo%3Aelastic%2Fkibana%20EcsFlat&type=code)**
---------
Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Close https://github.com/elastic/kibana/issues/181992
## Summary
First iteration of a CLI to capture an OAS snapshot.
## How to test
Run `node ./scripts/capture_oas_snapshot.js --update --include-path
/api/status` and see result in `oas_docs/bundle.json`.
If you have the [bump CLI](https://www.npmjs.com/package/bump-cli)
installed you can preview the hosted output with `bump preview
./oas_docs/bundle.json`
## Notes
* Added ability to filter by `version`, `access` (public/internal) and
excluding paths explicitly to the OAS generation lib
* Follows the same general pattern as our other "capture" CLIs like
`packages/kbn-check-mappings-update-cli`
* Result includes only `/api/status` for now, waiting for other paths to
add missing parts
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Relates to https://github.com/elastic/kibana/issues/183406.
## 📝 Summary
This PR creates a new plugin `data_quality` in order to register dataset
quality as a Stack management page under data section. For now there is
no reference to this new page in the sideNav in stateful or serverless.
In order to navigate to this new page you can use the url
`/app/management/data/data_quality`
Changes included in this PR:
- New plugin created
- Plugin registered in stack management, data section
- Dataset quality plugin is instantiated and the state is in sync with
URL
- Removed references to dataset quality in Logs explorer
## 🎥 Demo
501c9c47-4a1b-4f91-9be6-d022a821e88e
## 🙅🏼 Missing
- Dataset quality locator
- There are still references to logs explorer (table and flyout) that
will be handled in a follow up PR.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
It fixes#179633
Observability created a Comparator type/enum, when ResponseOps is
already exporting one and other rules using it.
The only difference is the wording of not in between [I put the two
types side by side to compare]
Currently, we import the one in triggers-actions-ui-plugin , and then
update the not in between to match our Comparator.
### Comparing the two enums:
## For reviewers 🧪
- Everything should work as expected: Alert flyout, Alert reason
message, Rule creation flyout, etc.
- I kept the `outside` comparator (replaced by `NOT BETWEEN`) for
backward compatibility
## Summary
As agreed in the recent ResponseOps/Security/Observability meeting, the
ResponseOps team will take ownership of the alerts grouping package
(formerly `@kbn/securitysolution-grouping`) in order to provide a
solution-agnostic API surface for future usages in Observability and
Stack rules.
This PR implements this transfer, while also renaming the package to
`@kbn/grouping` to reflect the change in scope.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This is a "proof of concept" for generating entity-centric indices for
the OAM. This exposes an API (`/api/entities`) for creating "asset
definitions" (`EntityDefinition`) that manages a transform and ingest
pipeline to produce documents into an index which could be used to
create a search experience or lookups for different services.
### Features
- Data schema agnostic, works with known schemas OR custom logs
- Supports defining multiple `identityFields` along with an
`identityTemplate` for formatting the `asset.id`
- Supports optional `identityFields` using `{ "field": "path-to-field",
"optional": true }` definition instead of a `string`.
- Supports defining key `metrics` with equations which are compatible
with the SLO product
- Supports adding `metadata` fields which will include multiple values.
- Supports `metadata` fields can be re-mapped to a new destination path
using `{ "source": "path-to-source-field", "limit": 1000, "destination":
"path-to-destination-in-output" }` definition instead of a `string`
- Supports adding `staticFields` which can also use template variables
- Support fine grain control over the frequency and sync settings for
the underlying transform
- Installs the index template components and index template settings for
the destination index
- Allow the user to configure the index patterns and timestamp field
along with the lookback
- The documents for each definition will be stored in their own index
(`.entities-observability.summary-v1.{defintion.id}`)
### Notes
- We are currently considering adding a historical index which will
track changes to the assets over time. If we choose to do this, the
summary index would remain the same but we'd add a second transform with
a group_by on the `definition.timestampField` and break the indices into
monthly indexes (configurable in the settings).
- We are looking into ways to add `firstSeenTimestamp`, this is a
difficult due to scaling issue. Essentially, we would need to find the
`minimum` timestamp for each entity which could be extremely costly on a
large datasets.
- There is nothing stopping you from creating an asset definition that
uses the `.entities-observability.summary-v1.*` index pattern to create
summaries of summaries... it can be very "meta".
### API
- `POST /api/entities/definition` - Creates a new asset definition and
starts the indexing. See examples below.
- `DELETE /api/entities/definition/{id}` - Deletes the asset definition
along with cleaning up the transform, ingest pipeline, and deletes the
destination index.
- `POST /api/entities/definition/{id}/_reset` - Resets the transform,
ingest pipeline, and destination index. This is useful for upgrading
asset definitions to new features.
## Example Definitions and Output
Here is a definition for creating services for each of the custom log
sources in the `fake_stack` dataset from `x-pack/packages/data-forge`.
```JSON
POST kbn:/api/entities/definition
{
"id": "admin-console-logs-service",
"name": "Services for Admin Console",
"type": "service",
"indexPatterns": ["kbn-data-forge-fake_stack.*"],
"timestampField": "@timestamp",
"lookback": "5m",
"identityFields": ["log.logger"],
"identityTemplate": "{{log.logger}}",
"metadata": [
"tags",
"host.name"
],
"metrics": [
{
"name": "logRate",
"equation": "A / 5",
"metrics": [
{
"name": "A",
"aggregation": "doc_count",
"filter": "log.level: *"
}
]
},
{
"name": "errorRate",
"equation": "A / 5",
"metrics": [
{
"name": "A",
"aggregation": "doc_count",
"filter": "log.level: \"ERROR\""
}
]
}
]
}
```
Which produces:
```JSON
{
"host": {
"name": [
"admin-console.prod.020",
"admin-console.prod.010",
"admin-console.prod.011",
"admin-console.prod.001",
"admin-console.prod.012",
"admin-console.prod.002",
"admin-console.prod.013",
"admin-console.prod.003",
"admin-console.prod.014",
"admin-console.prod.004",
"admin-console.prod.015",
"admin-console.prod.016",
"admin-console.prod.005",
"admin-console.prod.017",
"admin-console.prod.006",
"admin-console.prod.018",
"admin-console.prod.007",
"admin-console.prod.019",
"admin-console.prod.008",
"admin-console.prod.009"
]
},
"entity": {
"latestTimestamp": "2024-05-10T22:04:51.481Z",
"metric": {
"logRate": 37.4,
"errorRate": 1
},
"identity": {
"log": {
"logger": "admin-console"
}
},
"id": "admin-console",
"indexPatterns": [
"kbn-data-forge-fake_stack.*"
],
"definitionId": "admin-console-logs-service"
},
"event": {
"ingested": "2024-05-10T22:05:51.955691Z"
},
"tags": [
"infra:admin-console"
]
}
```
Here is an example of a definition for APM Services:
```JSON
POST kbn:/api/entities/definition
{
"id": "apm-services",
"name": "Services for APM",
"type": "service",
"indexPatterns": ["logs-*", "metrics-*"],
"timestampField": "@timestamp",
"lookback": "5m",
"identityFields": ["service.name", "service.environment"],
"identityTemplate": "{{service.name}}:{{service.environment}}",
"metadata": [
"tags",
"host.name"
],
"metrics": [
{
"name": "latency",
"equation": "A",
"metrics": [
{
"name": "A",
"aggregation": "avg",
"field": "transaction.duration.histogram"
}
]
},
{
"name": "throughput",
"equation": "A / 5",
"metrics": [
{
"name": "A",
"aggregation": "doc_count"
}
]
},
{
"name": "failedTransRate",
"equation": "A / B",
"metrics": [
{
"name": "A",
"aggregation": "doc_count",
"filter": "event.outcome: \"failure\""
},
{
"name": "B",
"aggregation": "doc_count",
"filter": "event.outcome: *"
}
]
}
]
}
```
Which produces:
```JSON
{
"host": {
"name": [
"simianhacker's-macbook-pro"
]
},
"entity": {
"latestTimestamp": "2024-05-10T21:38:22.513Z",
"metric": {
"latency": 615276.8812785388,
"throughput": 50.6,
"failedTransRate": 0.0091324200913242
},
"identity": {
"service": {
"environment": "development",
"name": "admin-console"
}
},
"id": "admin-console:development",
"indexPatterns": [
"logs-*",
"metrics-*"
],
"definitionId": "apm-services"
},
"event": {
"ingested": "2024-05-10T21:39:33.636225Z"
},
"tags": [
"_geoip_database_unavailable_GeoLite2-City.mmdb"
]
}
```
### Getting Started
The easiest way to get started is to use the`kbn-data-forge` config
below. Save this YAML to `~/Desktop/fake_stack.yaml` then run `node
x-pack/scripts/data_forge.js --config ~/Desktop/fake_stack.yaml`. Then
create a definition using the first example above.
```YAML
---
elasticsearch:
installKibanaUser: false
kibana:
installAssets: true
host: "http://localhost:5601/kibana"
indexing:
dataset: "fake_stack"
eventsPerCycle: 50
reduceWeekendTrafficBy: 0.5
schedule:
# Start with good events
- template: "good"
start: "now-1d"
end: "now-20m"
eventsPerCycle: 50
randomness: 0.8
- template: "bad"
start: "now-20m"
end: "now-10m"
eventsPerCycle: 50
randomness: 0.8
- template: "good"
start: "now-10m"
end: false
eventsPerCycle: 50
randomness: 0.8
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Moved the TryInConsoleButton from the search api panels package to its
own package. This should make it easier to import and encourage usage in
more parts of Kibana even outside of search.
## Summary
Something to get us started on the AST fun. At least until we contribute
ES|QL support to https://github.com/fkling/astexplorer :)
82c482b7-cd61-4440-b723-84f863c1b596
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## 📓 Summary
This work is just the extraction of a utility hook implemented in the
`infra` plugin for re-usability, as there are some scenarios where it
becomes handy in the logs explorer plugin too.
For improved reusability in future, I extracted the useBoolean hook into
a react-hooks package, where additional stateful utility hooks can go in
future.
This also replaces the current usage of the hook in the `infra` plugin
and is used to refactor part of the logs_explorer top nav.
---------
Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Revives this https://github.com/elastic/kibana/pull/181969
To do so, I had to create a new package `search-types` and move the
types I need there.
The Discovery team can take it from here.
Note: It also does a cleanup on the types I move, some of them were
declared twice.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## 📓 Summary
Closes#181528Closes#181976
**N.B.** This work was initially reviewed on a separate PR at
https://github.com/tonyghiani/kibana/pull/1.
This implementation aims to have a stateful layer that allows the
management of dependencies between Discover and other plugins, reducing
the need for a direct dependency.
Although the initial thought was to have a plugin to register features
for Discover, there might be other cases in future where we need to
prevent cyclic dependencies.
With this in mind, I created the plugin as a more generic solution to
hold stateful logic as a communication layer for Discover <-> Plugins.
## Discover Shared
Based on some recurring naming in the Kibana codebase, `discover_shared`
felt like the right place for owning these dependencies and exposing
Discover functionalities to the external world.
It is initially pretty simple and only exposes a registry for the
Discover features, but might be a good place to implement other upcoming
concepts related to Discover.
Also, this plugin should ideally never depend on other solution plugins
and keep its dependencies to a bare minimum of packages and core/data
services.
```mermaid
flowchart TD
A(Discover) -- Get --> E[DiscoverShared]
B(Logs Explorer) -- Set --> E[DiscoverShared]
C(Security) -- Set --> E[DiscoverShared]
D(Any app) -- Set --> E[DiscoverShared]
```
## DiscoverFeaturesService
This service initializes and exposes a strictly typed registry to allow
consumer apps to register additional features and Discover and retrieve
them.
The **README** file explains a real use case of when we'd need to use it
and how to do that step-by-step.
Although it introduces a more nested folder structure, I decided to
implement the service as a client-service and expose it through the
plugin lifecycle methods to provide the necessary flexibility we might
need:
- We don't know yet if any of the features we register will be done on
the setup/start steps, so having the registry available in both places
opens the door to any case we face.
- The service is client-only on purpose. My opinion is that if we ever
need to register features such as server services or anything else, it
should be scoped to a similar service dedicated for the server lifecycle
and its environment.
It should never be possible to register the ObsAIAssistant
presentational component from the server, as it should not be permitted
to register a server service in the client registry.
A server DiscoverFeaturesService is not required yet for any feature, so
I left it out to avoid overcomplicating the implementation.
## FeaturesRegistry
To have a strictly typed utility that suggests the available features on
a registry and adheres to a base contract, the registry exposed on the
DiscoverFeaturesService is an instance of the `FeaturesRegistry` class,
which implements the registration/retrieval logic such that:
- If a feature is already registered, is not possible to override it and
an error is thrown to notify the user of the collision.
- In case we need to react to registry changes, is possible to subscribe
to the registry or obtain it as an observable for more complex
scenarios.
The FeaturesRegistry already takes care of the required logic for the
registry, so that `DiscoverFeaturesService`is left with the
responsibility of instantiating/exposing an instance and provide the set
of allowed features.
---------
Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Davis McPhee <davismcphee@hotmail.com>
Create an empty assets data access plugin so we can start registering
the services(APIs) that will be used to fetch assets type documents.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
### Changes
1. adds `hidePanelChrome` parameter to `ReactEmbeddableRenderer`. When
true, embeddable is rendered without `PresentationPanel` wrapper
2. Removes `embeddable-explorer` plugin
3. Moves Embeddable developer example into embeddable_examples plugin
4. Creates new examples that demonstrate how to use
`ReactEmbeddableRenderer`
<img width="600" alt="Screenshot 2024-04-23 at 5 19 18 PM"
src="5167a2a4-1968-42e6-92f7-4577c9cda3c6">
Follow-up work to narrow scope of this PR
1. add key concepts to embeddable overview
2. add "Register new embeddable type" tab that details how to create a
new embeddable and shows how you can add embeddable examples to
dashboard
3. group embeddable examples into a single item in "Add panel" menu - to
show best practices of how to keep menu clean
4. remove class based example embeddables
### Test instructions
1. start kibana with `yarn start --run-examples`
5. Open kibana menu and click "Developer examples"
6. Click "Embeddables" card and run examples
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fix https://github.com/elastic/kibana/issues/178932
- Introduce the new `userProfile` core service, both on the browser and
server-side.
- Have the security plugin register its API to Core for re-exposition
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
* Creates a new package - ` @kbn/inference_integration_flyout`
* Above package has implementation for adding new inference endpoint id,
including Elastic, third party integration and instructions to upload
via Eland python client, in a flyout.
* The above package is used for supporting semantic_text feature and is
accessed via`add a new inference endpoint` button
## Screen Recording
dbd36634-bd4a-49f1-b1d5-d7b6c90444bc
### Checklist
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Introduces a new package for generating OAS from Kibana's routers. This
first iteration includes:
* E2E conversion of Core's `Router` and `CoreVersionedRouter` routes
into a single OAS document (not written to disk or shared anywhere
yet...)
* Support for
[`$ref`](https://swagger.io/docs/specification/using-ref/?sbsearch=%24ref)
by introducing the `meta.id` field `@kbn/config-schema`'s base type.
This is intended to be used only response/request schemas initially.
## TODO
- [x] More unit tests
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
