Implements chat & function calling for the Observability AI Assistant.
The APM related changes are due to a correction in the `toBooleanRt`
type.
Code in `x-pack/observability_ai_assistant` has been reviewed via
feature branches.
---------
Co-authored-by: Coen Warmer <coen.warmer@gmail.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Clint Andrew Hall <clint@clintandrewhall.com>
closes [#157505](https://github.com/elastic/kibana/issues/157505)
fixes [#157740](https://github.com/elastic/kibana/issues/157740)
## Summary
This PR adds strict payload validation to `inventory_views` endpoint. I
tried to make the types consistent between sever and frontend code and
because of that more changes had to be made. I also refactored the
`toolbar_control` component, decoupling them from Inventory View and
Metrics Explorer View types
### How to test
- Call the endpoint below trying to use invalid values. see
[here](https://github.com/elastic/kibana/pull/160852/files#diff-058b21e249ebbe2795d450d07025d8904a58cfb07a97979e85975f87e931ffceR133-R143).
```bash
[POST|PUT] kbn:/api/infra/inventory_views
{
"attributes": {
"metric": {
"type": [TYPE]
},
"sort": {
"by": "name"
"direction": "desc"
},
"groupBy": [],
"nodeType": "host",
"view": "map",
"customOptions": [],
"customMetrics": [],
"boundsOverride": {
"max": 1,
"min": 0
},
"autoBounds": true,
"accountId": "",
"region": "",
"autoReload": false,
"filterQuery": {
"expression": "",
"kind": "kuery"
},
"legend": {
"palette": "cool",
"steps": 10,
"reverseColors": false
},
"timelineOpen": false,
"name": "test-uptime"
}
}
```
- Set up a local Kibana instance
- Navigate to `Infrastructure`
- In the UI, use the Saved View feature and try different field
combinations
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## 📓 Summary
Closes https://github.com/elastic/observability-dev/issues/2655
This PR introduces a customized log consumption experience in the
Discover plugin. By leveraging the new `discover_log_explorer` plugin
and utilizing the `discover.customize` functionality, we have curated a
more tailored user experience.
The key feature of this implementation is the `DatasetSelector`
component, which replaces the original Discover `DataViewPicker`. It
handles the retrieval, rendering, and navigation of integrations and
data streams related to logs, providing an improved user interface.
This PR involves significant development efforts, including the creation
of the `discover_log_explorer` plugin, implementation of services, state
machines, custom hooks, and enhancements to presentational components.
The following overview will help reviewers understand the
responsibilities of each component in this implementation.
d725b699-452e-4718-8189-8dc1fab4d044
## DatasetsService & DatasetsClient
The DatasetsService is introduced, a crucial component that mediates
access to the newly implemented DatasetsClient. During the plugin's
lifecycle, the DatasetsService exposes a client property through its
start() method, providing convenient access to a DatasetsClient
instance.
The DatasetsClient is responsible for abstracting the data fetching
process for two endpoints: the integrations endpoint and the data
streams listing endpoint. These endpoints are utilized to populate the
selector options in the user interface. To facilitate this, the
DatasetsClient exposes the findIntegrations and findDatasets methods,
which handle the respective data fetching.
## Discover Customization
The critical part of this work consists of where the customization is
applied.
Inside the `public/plugin.tsx`, we lazy load and create, injecting the
required dependencies, the `CustomDatasetSelector`, which already
encapsulates all the logic required to make the selector work with the
external APIs.
We kept separating the data fetching logic from how the selector works,
and all the data and events are passed into the UI component with
properties.
```ts
discover.customize(
DISCOVER_LOG_EXPLORER_PROFILE_ID,
({ customizations, stateContainer }) => {
customizations.set({
id: 'search_bar',
CustomDataViewPicker: createLazyCustomDatasetSelector({
datasetsClient: datasetsService.client,
stateContainer,
}),
});
...
```
## Data fetching state machines & custom hooks
To handle the data fetching of integrations and unmanaged data streams,
we created two different state machines to separately handle the related
action for each dataset, such as remote search, in-memory search, error
handling etc.
### Integration machine and useIntegrations
The integrations state machine handles automatic data fetching of the
resources and additionally provides transitions for loading more
integrations, searching integrations by HTTP request, searching locally
into integration streams, and all the related loading and error handling
states.
It is then interpreted inside the `useIntegrations` custom hook, which
exposes the fetched data and handlers for all the above-mentioned
actions.
<img width="1975" alt="Screenshot 2023-05-30 at 09 44 42"
src="6daeca9f-826d-4a0f-bd90-eb4826ed1bde">
### Datasets machine and useDatasets
Similar to the integrations state machine, but simplified since the data
streams search can only happen with HTTP requests and there is no
pagination that requires to handle the load of more entries.
It is interpreted inside the `useDatasets` custom hook, which also
exposes the fetched data and handlers for the available actions.
<img width="1692" alt="Screenshot 2023-05-30 at 09 45 11"
src="5f9690e2-4e8f-439e-9ffd-f3b34cf3eaf5">
## DatasetSelector
The `DatasetSelector` component contains all the logic that manages the
navigation and searches across the different panels that render
integrations, integrations' streams or unmanaged streams.
As the datasets come from different APIs or are performed in-memory, the
search work follow this logic:
- When listing the integrations list (first level of the
`EuiContextMenu`), the search is done with an HTTP request.
- When listing the data streams list for a specific integration (second
level of the `EuiContextMenu`), the search is done in-memory, filtering
and sorting directly in the client.
- When listing the unmanaged data streams list (second level of the
`EuiContextMenu`), the search is done again with an HTTP request.
To handle these possible user journeys correctly without side effects,
we created another state machine and exposed its actions with an
internal `useDatasetSelector` custom hook.
<img width="1978" alt="Screenshot 2023-05-30 at 09 46 04"
src="84aa4247-c65d-40de-9eb6-6117bee731f8">
## Next steps
This component will change quite a lot until we won't get to a final
design. As soon as a first solid mvp is defined for production, a
complete test for the component will be implemented, among with a more
generic functional test for the core customization features.
---------
Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
Co-authored-by: Felix Stürmer <weltenwort@users.noreply.github.com>
closes https://github.com/elastic/kibana/issues/154147
## Summary
This PR introduces `io-ts` runtime validation to asset manager
endpoints.
The parameters will be validated as follows:
- `maxDistance`: range from 1 to 5
- `size`: range from 1 to 100
- `type`: `k8s.pod` or `k8s.cluster` or `k8s.node`
- `relation`: `ancestors` or `descendants` or `references`
- any date parameter: Date ISO or DateMath formats
### How to test this PR
- enable `xpack.assetManager.alphaEnabled` in kibana.dev.yml
- Call the endpoints below and try to pass invalid parameters where
they're applicable:
- `GET
kbn:/api/asset-manager/assets?from=2023-03-25T17:44:44.000Z&to=2023-03-25T18:44:44.000Z`
- `GET
kbn:/api/asset-manager/assets/diff?aFrom=2022-02-07T00:00:00.000Z&aTo=2022-02-07T01:30:00.000Z&bFrom=2022-02-07T01:00:00.000Z&bTo=2022-02-07T02:00:00.000Z`
- `GET
kbn:/api/asset-manager/assets/related?ean=k8s.node:node-101&relation=ancestors&maxDistance=1&from=2023-04-18T13:10:13.111Z&to=2023-04-18T15:10:13.111Z`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## 📓 Summary
Part of #152617Closes#155158
This PR implements the CRUD endpoints for the inventory views.
Following the approach used for the LogViews service, it exposes a
client that abstracts all the logic concerned to the `inventory-view`
saved objects.
It also follows the guideline provided for [Versioning
interfaces](https://docs.elastic.dev/kibana-dev-docs/versioning-interfaces)
and [Versioning HTTP
APIs](https://docs.elastic.dev/kibana-dev-docs/versioning-http-apis),
preparing for the serverless.
## 🤓 Tips for the reviewer
You can open the Kibana dev tools and play with the following snippet to
test the create APIs, or you can perform the same requests with your
preferred client:
```
// Get all
GET kbn:/api/infra/inventory_views
// Create one
POST kbn:/api/infra/inventory_views
{
"attributes": {
"name": "My inventory view"
}
}
// Get one
GET kbn:/api/infra/inventory_views/<switch-with-id>
// Update one
PUT kbn:/api/infra/inventory_views/<switch-with-id>
{
"attributes": {
"name": "My inventory view 2"
}
}
// Delete one
DELETE kbn:/api/infra/inventory_views/<switch-with-id>
```
## 👣 Next steps
- Replicate the same logic for the metrics explorer saved object
- Create a client-side abstraction to consume the service
- Update the existing react custom hooks to consume the endpoint
---------
Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Dearest Reviewers 👋
I've been working on this branch with @mistic and @tylersmalley and
we're really confident in these changes. Additionally, this changes code
in nearly every package in the repo so we don't plan to wait for reviews
to get in before merging this. If you'd like to have a concern
addressed, please feel free to leave a review, but assuming that nobody
raises a blocker in the next 24 hours we plan to merge this EOD pacific
tomorrow, 12/22.
We'll be paying close attention to any issues this causes after merging
and work on getting those fixed ASAP. 🚀
---
The operations team is not confident that we'll have the time to achieve
what we originally set out to accomplish by moving to Bazel with the
time and resources we have available. We have also bought ourselves some
headroom with improvements to babel-register, optimizer caching, and
typescript project structure.
In order to make sure we deliver packages as quickly as possible (many
teams really want them), with a usable and familiar developer
experience, this PR removes Bazel for building packages in favor of
using the same JIT transpilation we use for plugins.
Additionally, packages now use `kbn_references` (again, just copying the
dx from plugins to packages).
Because of the complex relationships between packages/plugins and in
order to prepare ourselves for automatic dependency detection tools we
plan to use in the future, this PR also introduces a "TS Project Linter"
which will validate that every tsconfig.json file meets a few
requirements:
1. the chain of base config files extended by each config includes
`tsconfig.base.json` and not `tsconfig.json`
1. the `include` config is used, and not `files`
2. the `exclude` config includes `target/**/*`
3. the `outDir` compiler option is specified as `target/types`
1. none of these compiler options are specified: `declaration`,
`declarationMap`, `emitDeclarationOnly`, `skipLibCheck`, `target`,
`paths`
4. all references to other packages/plugins use their pkg id, ie:
```js
// valid
{
"kbn_references": ["@kbn/core"]
}
// not valid
{
"kbn_references": [{ "path": "../../../src/core/tsconfig.json" }]
}
```
5. only packages/plugins which are imported somewhere in the ts code are
listed in `kbn_references`
This linter is not only validating all of the tsconfig.json files, but
it also will fix these config files to deal with just about any
violation that can be produced. Just run `node scripts/ts_project_linter
--fix` locally to apply these fixes, or let CI take care of
automatically fixing things and pushing the changes to your PR.
> **Example:** [`64e93e5`
(#146212)](64e93e5806)
When I merged main into my PR it included a change which removed the
`@kbn/core-injected-metadata-browser` package. After resolving the
conflicts I missed a few tsconfig files which included references to the
now removed package. The TS Project Linter identified that these
references were removed from the code and pushed a change to the PR to
remove them from the tsconfig.json files.
## No bazel? Does that mean no packages??
Nope! We're still doing packages but we're pretty sure now that we won't
be using Bazel to accomplish the 'distributed caching' and 'change-based
tasks' portions of the packages project.
This PR actually makes packages much easier to work with and will be
followed up with the bundling benefits described by the original
packages RFC. Then we'll work on documentation and advocacy for using
packages for any and all new code.
We're pretty confident that implementing distributed caching and
change-based tasks will be necessary in the future, but because of
recent improvements in the repo we think we can live without them for
**at least** a year.
## Wait, there are still BUILD.bazel files in the repo
Yes, there are still three webpack bundles which are built by Bazel: the
`@kbn/ui-shared-deps-npm` DLL, `@kbn/ui-shared-deps-src` externals, and
the `@kbn/monaco` workers. These three webpack bundles are still created
during bootstrap and remotely cached using bazel. The next phase of this
project is to figure out how to get the package bundling features
described in the RFC with the current optimizer, and we expect these
bundles to go away then. Until then any package that is used in those
three bundles still needs to have a BUILD.bazel file so that they can be
referenced by the remaining webpack builds.
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
closes#146517
This PR expands the current `includeTimeseries` parameter behaviour, by
not using the `date_histogram` in the `composite` aggregation when it is
set to `false`.
This way, not only the API won't return a `timeseries` object, but the
query will calculate the aggregations on ES side as opposed to when
`includeTimeseries` is true, that does an extra `avg` and `max`
calculation on TS side using the histogram buckets.
The main motivation behind this change is to run the snapshot query
without the need of returning buckets, to avoid the max bucket error.
### How to test
- Enable the system module on metricbeat
- Start a local Kibana
- Start metricbeat
- Explore `Infrastructure > Metrics`, `Infrastructure > Inventory` and
`Infrastructure > Hosts` and play with the filters
#### cURL
##### Run a query with Composite aggregation
###### With timeseries
```bash
curl -XPOST -f 'http://0.0.0.0:5601/ftw/api/metrics/snapshot' \
-H 'kbn-xsrf:localhost' \
-H 'Content-Type: application/json' \
--data-raw '{"filterQuery":"{\"bool\":{\"must\":[],\"filter\":[],\"should\":[],\"must_not\":[]}}","metrics":[{"type":"memory"}],"groupBy":[],"nodeType":"host","sourceId":"default","accountId":"","region":"","timerange":{"from":1670859470896,"to":1670861030896,"interval":"1m","ignoreLookback":true},"includeTimeseries":true}' \
-u elastic:changeme
```
###### Without timeseries
```bash
curl -XPOST -f 'http://0.0.0.0:5601/ftw/api/metrics/snapshot' \
-H 'kbn-xsrf:localhost' \
-H 'Content-Type: application/json' \
--data-raw '{"filterQuery":"{\"bool\":{\"must\":[],\"filter\":[],\"should\":[],\"must_not\":[]}}","metrics":[{"type":"memory"}],"groupBy":[],"nodeType":"host","sourceId":"default","accountId":"","region":"","timerange":{"from":1670859470896,"to":1670861030896,"interval":"1m","ignoreLookback":true},"includeTimeseries":false}' \
-u elastic:changeme
```
##### Run a query with just Date Histogram
###### With timeseries
```bash
curl -XPOST -f 'http://0.0.0.0:5601/ftw/api/metrics/snapshot' \
-H 'kbn-xsrf:localhost' \
-H 'Content-Type: application/json' \
--data-raw '{"filterQuery":"{\"bool\":{\"must\":[],\"filter\":[],\"should\":[],\"must_not\":[]}}","metrics":[{"type":"memory"}],"groupBy":null,"nodeType":"host","sourceId":"default","accountId":"","region":"","timerange":{"from":1670859470896,"to":1670861030896,"interval":"1m","ignoreLookback":true},"includeTimeseries":true}' \
-u elastic:changeme
```
##### Without timeseries
```bash
curl -XPOST -f 'http://0.0.0.0:5601/ftw/api/metrics/snapshot' \
-H 'kbn-xsrf:localhost' \
-H 'Content-Type: application/json' \
--data-raw '{"filterQuery":"{\"bool\":{\"must\":[],\"filter\":[],\"should\":[],\"must_not\":[]}}","metrics":[{"type":"memory"}],"groupBy":null,"nodeType":"host","sourceId":"default","accountId":"","region":"","timerange":{"from":1670859470896,"to":1670861030896,"interval":"1m","ignoreLookback":true},"includeTimeseries":false}' \
-u elastic:changeme
```
### For maintainers
The `logRate` metric type <b>doesn't</b> work without being in a date
histogram, because it uses [cumulative
sum](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-cumulative-sum-aggregation.html).
Therefore `includeTimeseries` false in a `composite` aggregation will
return a 400 error. I haven't found anywhere that would make a request
like below.
#### 400 error
```bash
curl -XPOST -f 'http://0.0.0.0:5601/ftw/api/metrics/snapshot' \
-H 'kbn-xsrf:localhost' \
-H 'Content-Type: application/json' \
--data-raw '{"filterQuery":"{\"bool\":{\"must\":[],\"filter\":[],\"should\":[],\"must_not\":[]}}","metrics":[{"type":"logRate"}],"groupBy":[],"nodeType":"host","sourceId":"default","accountId":"","region":"","timerange":{"from":1670859470896,"to":1670861030896,"interval":"1m","ignoreLookback":true},"includeTimeseries":false}' \
-u elastic:changeme
```
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Closes [#145238](https://github.com/elastic/kibana/issues/145238)
## Summary
These changes add validation to the Metric Indices passed into the
Metrics settings page.
New validation is added both in the UI and in the endpoint, performing
the following checks:
- Index pattern is not an empty string
- Index pattern does not contain empty spaces (start, middle, end) (the
pattern is not trimmed)
- Index pattern does not contain empty entries, comma-separated values
should have an acceptable value.
In case the value is not valid, the UI will render an appropriate error
message.
If the `PATCH /api/metrics/source/{sourceId}` request to update the
value is manually sent with an invalid value, the server will respond
with a 400 status code and an error message.
Also, for retro compatibility and to not block the user when the
configuration can't be successfully retrieved, in case of internal error
the `GET /api/metrics/source/{sourceId}` will return a 404 and on the
UI, instead of rendering a blank page, the user will see the empty form
and will be able to re-insert the right values.
## Testing
Navigate to `Inventory`-> Click on `Settings` on the topbar -> Start
writing different metric indices in the Metric Indices field.
### Editing Metric Indices validation
https://user-images.githubusercontent.com/34506779/203763021-0f4d8926-ffa4-448a-a038-696732158f4e.mov
### Missing/Broken configuration response
https://user-images.githubusercontent.com/34506779/203763120-ffc91cd3-9bf4-43da-a04f-5561ceabf591.mov
Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* [packages] add kibana.jsonc files
* auto-migrate to kibana.jsonc
* support interactive pkg id selection too
* remove old codeowners entry
* skip codeowners generation when .github/CODEOWNERS doesn't exist
* fall back to format validation if user is offline
* update question style
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* refact(NA): apply root_input_dir=src to each already created pkg
* refact(NA): update package generator
* fix(NA): correctly use rootDir
* fix(NA): use root input dir on latest introduced pkgs for jsts_transpiler macro
* chore(NA): merge with main
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [type-summarizer] reimplement for broader support
* Enable sourceMaps in all packages
* include naming collision in summarizePackage test
* fix readmes
* remove unnecessary transient dependency
* remove code that was commented out
* remove outdated todo comment
* ensure errors triggered by untyped-exports are ligible
* remove unused import
* break out snippet generation from AstIndexer
* refactor several massive files into smaller pieces and add more inline docs
* fix typos
* update jest snapshots
* add sections to readme that points people to the useful parts of the source code along with a high-level overview of how the type-summarizer works
* remove --dump flag, it doesn't work
* use decName instead of calling names.get a second time
* include `export` as invalid name
* [build_ts_refs] improve caches, allow building a subset of projects
* cleanup project def script and update refs in type check script
* rename browser_bazel config to avoid kebab-case
* remove execInProjects() helper
* list references for tsconfig.types.json for api-extractor workload
* disable composite features of tsconfig.types.json for api-extractor
* set declaration: true to avoid weird debug error
* fix jest tests
Co-authored-by: spalger <spalger@users.noreply.github.com>
* [APM] @kbn/typed-router-config
* [APM] typed route config
* Breadcrumbs, wildcards
* Migrate settings, home
* Migrate part of service detail page
* Migrate remaining routes, tests
* Set maxWorkers for precommit script to 4
* Add jest types to tsconfigs
* Make sure transaction distribution data is fetched
* Fix typescript errors
* Remove usage of react-router's useParams
* Add route() utility function
* Don't use ApmServiceContext for alert flyouts
* Don't add onClick handler for breadcrumb
* Clarify ts-ignore
* Remove unused things
* Update documentation
* Use useServiceName() in ServiceMap component
* Set up Observability rule APIs
* Populate alerts table with data from API
* Move field map types/utils to common
* Format reason/link in alert type
* Format reason/link in alert type
* Fix issues with tsconfigs
* Storybook cleanup for example alerts
* Use `MemoryRouter` in the stories and `useHistory` in the component to get the history
* Replace examples with ones from "real" data
* Use `() => {}` instead of `jest.fn()` in mock registry data
* Store/display evaluations, add active/recovered badge
* Some more story fixes
* Decode rule data with type from owning registry
* Use transaction type/environment in link to app
* Fix type issues
* Fix API tests
* Undo changes in task_runner.ts
* Remove Mutable<> wrappers for field map
* Remove logger.debug calls in alerting es client
* Add API test for recovery of alerts
* Revert changes to src/core/server/http/router
* Use type imports where possible
* Update limits
* Set limit to 100kb
Co-authored-by: Nathan L Smith <smith@nlsmith.com>
