## Summary
Closes https://github.com/elastic/kibana/issues/66716
Improves code editors in Anomaly detection, Data frame analytics and
Transform wizards with autocomplete, data types validation and inline
documentation from elasticsearch specification.
Adds a package with JSON schemas extracted from the [openapi output of
elasticsearch-specification](https://github.com/elastic/elasticsearch-specification/tree/main/output/openapi).
Schema file is generated per editor/endpoint, keeping only relevant
components.
To test the script locally, execute
```
yarn run jsonSchema
```
from `/x-pack/packages/ml/json_schemas`.
By default it assumes that your `elasticsearch-specification` folder is
located next to the `kibana` repo, but you can also provide a path to
`openapi` file as a parameter, e.g. `yarn run jsonSchema
/Users/my_user/dev/elasticsearch-specification/output/openapi/elasticsearch-serverless-openapi.json`
#### How JSON files are served
JSON files are asynchronously imported at the React component level and
passed to the `CodeEditor` instances on mount.
Initially I tried different approaches to take advantage of CDN, but
unfortunately it didn't work out:
##### Using static assets
I made an attempt to retrieve a URL to the JSON schema file as a static
asset using Kibana service
```typescript
const schemaJsonAsset = http?.staticAssets.getPluginAssetHref('my_schema.json') ?? '';
```
and passing it as part of the schema definition, but the browser was
blocking a request 🤔
##### Using raw loader
Approach with a raw loader also didn't succeed.
```typescript
import mySchema from '!!raw-loader!./my_schema.json';
```
<details>
<summary>See error </summary>
```
ERROR in
./public/app/sections/create_transform/components/advanced_pivot_editor/my_schema.json
(/Users/dimaarnautov/Repos/kibana/node_modules/raw-loader/dist/cjs.js!./public/app/sections/create_transform/components/advanced_pivot_editor/my_schema.json)
│ Module parse failed: Unexpected token 'e', "export def"... is not
valid JSON while parsing 'export default "{\n \"type\": \"object\'
│ File was processed with these loaders:
│ * ../../../node_modules/raw-loader/dist/cjs.js
│ You may need an additional loader to handle the result of these
loaders.
│ SyntaxError: Unexpected token 'e', "export def"... is not valid JSON
while parsing 'export default "{\n \"type\": \"object\'
│ at JSON.parse (<anonymous>)
│ at parseJson
(/Users/dimaarnautov/Repos/kibana/node_modules/json-parse-better-errors/index.js:7:17)
│ at JsonParser.parse
(/Users/dimaarnautov/Repos/kibana/node_modules/webpack/lib/JsonParser.js:16:16)
│ at
/Users/dimaarnautov/Repos/kibana/node_modules/webpack/lib/NormalModule.js:482:32
│ at
/Users/dimaarnautov/Repos/kibana/node_modules/webpack/lib/NormalModule.js:358:12
│ at
/Users/dimaarnautov/Repos/kibana/node_modules/loader-runner/lib/LoaderRunner.js:373:3
│ at iterateNormalLoaders
(/Users/dimaarnautov/Repos/kibana/node_modules/loader-runner/lib/LoaderRunner.js:214:10)
│ at iterateNormalLoaders
(/Users/dimaarnautov/Repos/kibana/node_modules/loader-runner/lib/LoaderRunner.js:221:10)
│ at
/Users/dimaarnautov/Repos/kibana/node_modules/loader-runner/lib/LoaderRunner.js:236:3
│ at runSyncOrAsync
(/Users/dimaarnautov/Repos/kibana/node_modules/loader-runner/lib/LoaderRunner.js:130:11)
│ at iterateNormalLoaders
(/Users/dimaarnautov/Repos/kibana/node_modules/loader-runner/lib/LoaderRunner.js:232:2)
│ at
/Users/dimaarnautov/Repos/kibana/node_modules/loader-runner/lib/LoaderRunner.js:205:4
│ at
/Users/dimaarnautov/Repos/kibana/node_modules/webpack/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:85:15
│ at processTicksAndRejections (node:internal/process/task_queues:77:11)
```
</details>
### Checklist
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
Fixes https://github.com/elastic/kibana/issues/169771
Adds a new endpoint
`/internal/ml/trained_models/install_elastic_trained_model/:modelId`
which wraps the `putTrainedModel` call to start the download of the
elser model. It then reassigns the saved object's space to be `*`.
Also updates the saved object sync call to ensure any internal models
(ones which start with `.`) are assigned to the `*` space, if they've
needed syncing.
It is still possible for a user to reassign the spaces for an elser
model and get themselves into the situation covered described in
https://github.com/elastic/kibana/issues/169771.
In this situation, I believe the best we can do is suggest the user
adjusts the spaces via the stack management page.
At the moment a `Model already exists` error is displayed in a toast. In
a follow up PR we could catch this and show more information to direct
the user to the stack management page.
---------
Co-authored-by: Dima Arnautov <arnautov.dima@gmail.com>
## Summary
Adds a shared service for elastic curated models. The first use case is
to provide a default/recommended ELSER version based on the hardware of
the current cluster.
#### Why?
In 8.11 we'll provide a platform-specific version of the ELSER v2
alongside the portable one. At the moment several solutions refer to
ELSER for download/inference purposes with a `.elser_model_1` constant.
Starting 8.11 the model ID will vary, so using the `ElastcModels`
service allows retrieving the recommended version of ELSER for the
current cluster without any changes by solution teams in future
releases. It is still possible to request an older version of the model
if necessary.
#### Implementation
- Adds a new Kibana API endpoint `/trained_models/model_downloads` that
provides a list of model definitions, with the `default` and
`recommended` flags.
- Adds a new Kibana API endpoint `/trained_models/elser_config` that
provides an ELSER configuration based on the cluster architecture.
- `getELSER` method is exposed from the plugin `setup` server-side as
part of our shared services and plugin `start` client-side.
### Checklist
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
Uses the recently created [category validation
package](https://github.com/elastic/kibana/pull/161261) to perform
validation on the field selected for pattern analysis.
If the field is considered unsuitable for categorization, a warning
callout is displayed which lists the reasons it is unsuitable.
If the field is suitable, no callout is displayed.
Other changes:
- Adds the selected field to the URL state, so it is remembered on page
refresh.
- If no field is in the URL, it will look for a field called `message`
in the data view and auto select it.
- renames the ML route `/jobs/categorization_field_examples` to
`/jobs/categorization_field_validation` as it is a more accurate name
and it's consistent with the newly added route in AIOPs.
**Log Pattern Analysis page in ML**
**Log Pattern Analysis flyout in Discover**
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Previously, apidoc scripts were transpiled to TS and the JS files were
passed as parameters to the npm command. With the recent Kibana changes
related to `tsconfig`, the output structure has changed and caused
`apiDocs` to fail.
With this PR I introduced a scripts folder that utilizes
[setup_node_env](afb09ccf8a/src/setup_node_env/index.js (L10))
and allows us to omit `tsc` step.
Also, the `getTypeAtLocation` method started throwing an error in some
cases. For now, I've added an error handling and need to investigate
what's off.
