Rather than initializing the failure store right away when a new
data stream is created, we leave it empty and mark it for lazy
rollover. This results in the failure store only being initialized
(i.e. an index created) when a failure has actually occurred.
The exception to the rule is when a failure occurs while the data
stream is being auto-created. In that case, we do want to initialize
the failure store right away.
The cluster level dense vector stats returns the total number of dense vector indices globally including the replicas.
This commit fixes the total to only include the value count of the primary indices.
This change aligns with the docs stats which also reports the number of primary documents when used in cluster stats.
The indices stats API still reports granular results for replicas and primaries so the information is not lost.
This adds a test that generates
`docs/reference/esql/functions/kibana/inline_cast.json` which is a json
object who's keys are the names of valid inline casts and who's values
are the resulting data types.
I also moved one of the maps we use to make the inline casts to
`DataType`, which is a place where we want it.
Handle the "output memory allocator bytes" field if and only if it is present in the model size stats, as reported by the C++ backend.
This PR _must_ be merged prior to the corresponding `ml-cpp` one, to keep CI tests happy.
This adds `hamming` distances, the pop-count of `xor` byte vectors as a
first class citizen in painless.
For byte vectors, this means that we can compute hamming distances via
script_score (aka, brute-force).
The implementation of `hamming` is the same that is available in Lucene,
and when lucene 9.11 is merged, we should update our logic where
applicable to utilize it.
NOTE: this does not yet add hamming distance as a metric for indexed
vectors. This will be a future PR after the Lucene 9.11 upgrade.
* Add SparseVectorStats
* Update to use mappings in engine
* Update to be unique to primary shards
* Fix doc
* Fix null error in test
* Cleanup
* fix yaml
* remove comment
* add version to yaml
* Revert whitespace changes to stats doc
* fix yml test
* Checkstyle
* Fix NPE in test
* Update docs/changelog/108793.yaml
* Add link to sparse_vector field type in docs
* PR feedback
* Flesh out test a bit more
* PR feedback - alphabetize placement in docs
* Fix doc change
This adds a new quantization mechanism for HNSW and flat indices. Here
we add `int4` quantization via the `int4_hnsw` and `int4_flat` index
types. This quantization methodology further reduces the memory required
for fast HNSW, meaning that the memory required is 8x smaller than with
regular float32 values.
8x reduction means that 1M 1024 dimension vectors goes from requiring
3.8GB to 477MB.
Recall continues to stay steady, there is some reduction that is
recoverable via slightly oversampling and reranking. For example over
500k CohereV3 vectors, only 5 extra vectors are required to be gathered
to achieve over 0.98 recall in a brute-force scenario.
When you divide two integers or two longs we round towards 0. Like
Postgres or Java or Rust or C. Other systems, like MySQL or SPL or
Javascript or Python always produce a floating point number. We should
warn folks about this. It's genuinely unexpected for some folks. OTOH,
converting into a floating point number would be unexpected for other
folks. Oh well, let's document what we've got.
This is a refactoring of the internal logic that's used to translate
query-level into index-level field names for query APIs for
security entities (i.e. users, API Keys, and soon, roles).
The objective here is to have and reuse a single class to handle
all the translations for different security query APIs.
Fixing MvAppendTests CB exceptions by generating smaller geometries: the
test generates a lot of documents and the CB is too small for multiple
big shapes.
Fixes https://github.com/elastic/elasticsearch/issues/109409
This commit introduces a watchdog timer to monitor for long-running
tasks on network threads. If a network thread is active and has not made
progress for two consecutive ticks of the timer then the watchdog logs a
warning and a thread dump.
* Initial docs for put-inference for Mistral
* adds mistral embeddings to tutorial; add changelog
* update mistral text and dimensions
* fix mistral spelling error
* fix azure AI studio; fix Mistral label
* fix auto-formatted items
* change pipeline button back to azure openai
* put proper Azure AI Studio include in
* fix missing azure-openai; fix huggingface hidden
* fix mistral tab for reindex
* re-add Mistral service settings to put inference
