* Remove `es-test-dir` book-scoped variable
* Remove `plugins-examples-dir` book-scoped variable
* Remove `:dependencies-dir:` and `:xes-repo-dir:` book-scoped variables
- In `index.asciidoc`, two variables (`:dependencies-dir:` and `:xes-repo-dir:`) were removed.
- In `sql/index.asciidoc`, the `:sql-tests:` path was updated to fuller path
- In `esql/index.asciidoc`, the `:esql-tests:` path was updated idem
* Replace `es-repo-dir` with `es-ref-dir`
* Move `:include-xpack: true` to few files that use it, remove from index.asciidoc
Certain NLP models such as multilingual-e5-large require a prefix
string to be applied to the input text. For asymmetric tasks such as
information retrieval the prefix can be different when ingesting the
data and when searching it. For example text embedding model can
have a one prefix applied when the model is evaluated as part of an
knn search and a different prefix when ingesting documents.
* Added platform architecture field to TrainedModelMetadata and users of TrainedModelMetadata
* Added TransportVersions guarding for TrainedModelMetadata
* Prevent platform-specific models from being deployed on the wrong architecture
* Added logic to only verify node architectures for models which are platform specific
* Handle null platform architecture
* Added logging for the detection of heterogeneous platform architectures among ML nodes and refactoring to support this
* Added platform architecture field to TrainedModelConfig
* Stop platform-speficic model when rebalance occurs and the cluster has a heterogeneous architecture among ML nodes
* Added logic to TransportPutTrainedModelAction to return a warning response header when the model is paltform-specific and cannot be depoloyed on the cluster at that time due to heterogenous architectures among ML nodes
* Added MlPlatformArchitecturesUtilTests
* Updated Create Trained Models API docs to describe the new platform_architecture optional field.
* Updated/incremented InferenceIndexConstants
* Added special override to make models with linux-x86_64 in the model ID to be platform specific
Many multi-lingual and newer models use a tokenization scheme similar to
sentence-piece. This PR adds support for one of those tokenization
schemes, XLMRoBERTa.
The main changes are: - Support for xlm_roberta tokenization
configuration - Adding `scores` to the vocabulary document stored,
requiring that scores be the same size as the vocabulary - Adding a new
flat text file to resources that is the spm char normalizer.
Introduced in: #88439
* [ML] add text_similarity nlp task documentation
* Apply suggestions from code review
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
* Update docs/reference/ml/trained-models/apis/infer-trained-model.asciidoc
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
* Apply suggestions from code review
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
* Update docs/reference/ml/ml-shared.asciidoc
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
This commit adds initial windowing support for text_classification tasks.
Specifically, a user can now indicate a span (non-negative) indicating the tokenization windowing span when creating
sub-sequences.
Default value is span: -1 indicates that no windowing should take place.
This commit adds support for MPNet based models.
MPNet models differ from BERT style models in that:
- Special tokens are different
- Input to the model doesn't require token positions.
To configure an MPNet tokenizer for your pytorch MPNet based model:
```
"tokenization": {
"mpnet": {...}
}
```
The options provided to `mpnet` are the same as the previously supported `bert` configuration.
