mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-04-24 23:27:25 -04:00
70 lines
3.1 KiB
Text
70 lines
3.1 KiB
Text
[[re-ranking-overview]]
|
|
= Re-ranking
|
|
|
|
Many search systems are built on multi-stage retrieval pipelines.
|
|
|
|
Earlier stages use cheap, fast algorithms to find a broad set of possible matches.
|
|
|
|
Later stages use more powerful models, often machine learning-based, to reorder the documents.
|
|
This step is called re-ranking.
|
|
Because the resource-intensive model is only applied to the smaller set of pre-filtered results, this approach returns more relevant results while still optimizing for search performance and computational costs.
|
|
|
|
{es} supports various ranking and re-ranking techniques to optimize search relevance and performance.
|
|
|
|
[float]
|
|
[[re-ranking-two-stage-pipeline]]
|
|
== Two-stage retrieval pipelines
|
|
|
|
|
|
[float]
|
|
[[re-ranking-first-stage-pipeline]]
|
|
=== Initial retrieval
|
|
|
|
[float]
|
|
[[re-ranking-ranking-overview-bm25]]
|
|
==== Full-text search: BM25 scoring
|
|
|
|
{es} ranks documents based on term frequency and inverse document frequency, adjusted for document length.
|
|
https://en.wikipedia.org/wiki/Okapi_BM25[BM25] is the default statistical scoring algorithm in {es}.
|
|
|
|
[float]
|
|
[[re-ranking-ranking-overview-vector]]
|
|
==== Vector search: similarity scoring
|
|
|
|
Vector search involves transforming data into dense or sparse vector embeddings to capture semantic meanings, and computing similarity scores for query vectors.
|
|
Store vectors using `semantic_text` fields for automatic inference and vectorization or `dense_vector` and `sparse_vector` fields when you need more control over the underlying embedding model.
|
|
Query vector fields with `semantic`, `knn` or `sparse_vector` queries to compute similarity scores.
|
|
Refer to <<semantic-search,semantic search>> for more information.
|
|
|
|
[float]
|
|
[[re-ranking-ranking-overview-hybrid]]
|
|
==== Hybrid techniques
|
|
|
|
Hybrid search techniques combine results from full-text and vector search pipelines.
|
|
{es} enables combining lexical matching (BM25) and vector search scores using the <<rrf,Reciprocal Rank Fusion (RRF)>> algorithm.
|
|
|
|
[float]
|
|
[[re-ranking-overview-second-stage]]
|
|
=== Re-ranking
|
|
|
|
When using the following advanced re-ranking pipelines, first-stage retrieval mechanisms effectively generate a set of candidates.
|
|
These candidates are funneled into the re-ranker to perform more computationally expensive re-ranking tasks.
|
|
|
|
[float]
|
|
[[re-ranking-overview-semantic]]
|
|
==== Semantic re-ranking
|
|
|
|
<<semantic-reranking>> uses machine learning models to reorder search results based on their semantic similarity to a query.
|
|
Models can be hosted directly in your {es} cluster, or you can use <<inference-apis,inference endpoints>> to call models provided by third-party services.
|
|
Semantic re-ranking enables out-of-the-box semantic search capabilities on existing full-text search indices.
|
|
|
|
[float]
|
|
[[re-ranking-overview-ltr]]
|
|
==== Learning to Rank (LTR)
|
|
|
|
<<learning-to-rank>> is for advanced users.
|
|
Learning To Rank involves training a machine learning model to build a ranking function for your search experience that updates over time.
|
|
LTR is best suited for when you have ample training data and need highly customized relevance tuning.
|
|
|
|
include::semantic-reranking.asciidoc[]
|
|
include::learning-to-rank.asciidoc[]
|