mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-04-25 07:37:19 -04:00
136 lines
3.3 KiB
Text
136 lines
3.3 KiB
Text
[role="xpack"]
|
|
[[stream-inference-api]]
|
|
=== Stream inference API
|
|
|
|
.New API reference
|
|
[sidebar]
|
|
--
|
|
For the most up-to-date API details, refer to {api-es}/group/endpoint-inference[{infer-cap} APIs].
|
|
--
|
|
|
|
Streams a chat completion response.
|
|
|
|
IMPORTANT: The {infer} APIs enable you to use certain services, such as built-in {ml} models (ELSER, E5), models uploaded through Eland, Cohere, OpenAI, Azure, Google AI Studio, Google Vertex AI, Anthropic, Watsonx.ai, or Hugging Face.
|
|
For built-in models and models uploaded through Eland, the {infer} APIs offer an alternative way to use and manage trained models.
|
|
However, if you do not plan to use the {infer} APIs to use these models or if you want to use non-NLP models, use the <<ml-df-trained-models-apis>>.
|
|
|
|
|
|
[discrete]
|
|
[[stream-inference-api-request]]
|
|
==== {api-request-title}
|
|
|
|
`POST /_inference/<inference_id>/_stream`
|
|
|
|
`POST /_inference/<task_type>/<inference_id>/_stream`
|
|
|
|
|
|
[discrete]
|
|
[[stream-inference-api-prereqs]]
|
|
==== {api-prereq-title}
|
|
|
|
* Requires the `monitor_inference` <<privileges-list-cluster,cluster privilege>>
|
|
(the built-in `inference_admin` and `inference_user` roles grant this privilege)
|
|
* You must use a client that supports streaming.
|
|
|
|
|
|
[discrete]
|
|
[[stream-inference-api-desc]]
|
|
==== {api-description-title}
|
|
|
|
The stream {infer} API enables real-time responses for completion tasks by delivering answers incrementally, reducing response times during computation.
|
|
It only works with the `completion` and `chat_completion` task types.
|
|
|
|
The Chat completion {infer} API and the Stream {infer} API differ in their response structure and capabilities.
|
|
The Chat completion {infer} API provides more comprehensive customization options through more fields and function calling support.
|
|
If you use the `openai` service or the `elastic` service, use the Chat completion {infer} API.
|
|
|
|
[NOTE]
|
|
====
|
|
include::inference-shared.asciidoc[tag=chat-completion-docs]
|
|
====
|
|
|
|
[discrete]
|
|
[[stream-inference-api-path-params]]
|
|
==== {api-path-parms-title}
|
|
|
|
`<inference_id>`::
|
|
(Required, string)
|
|
The unique identifier of the {infer} endpoint.
|
|
|
|
|
|
`<task_type>`::
|
|
(Optional, string)
|
|
The type of {infer} task that the model performs.
|
|
|
|
|
|
[discrete]
|
|
[[stream-inference-api-request-body]]
|
|
==== {api-request-body-title}
|
|
|
|
`input`::
|
|
(Required, string or array of strings)
|
|
The text on which you want to perform the {infer} task.
|
|
`input` can be a single string or an array.
|
|
+
|
|
--
|
|
[NOTE]
|
|
====
|
|
Inference endpoints for the `completion` task type currently only support a
|
|
single string as input.
|
|
====
|
|
--
|
|
|
|
|
|
[discrete]
|
|
[[stream-inference-api-example]]
|
|
==== {api-examples-title}
|
|
|
|
The following example performs a completion on the example question with streaming.
|
|
|
|
|
|
[source,console]
|
|
------------------------------------------------------------
|
|
POST _inference/completion/openai-completion/_stream
|
|
{
|
|
"input": "What is Elastic?"
|
|
}
|
|
------------------------------------------------------------
|
|
// TEST[skip:TBD]
|
|
|
|
|
|
The API returns the following response:
|
|
|
|
|
|
[source,txt]
|
|
------------------------------------------------------------
|
|
event: message
|
|
data: {
|
|
"completion":[{
|
|
"delta":"Elastic"
|
|
}]
|
|
}
|
|
|
|
event: message
|
|
data: {
|
|
"completion":[{
|
|
"delta":" is"
|
|
},
|
|
{
|
|
"delta":" a"
|
|
}
|
|
]
|
|
}
|
|
|
|
event: message
|
|
data: {
|
|
"completion":[{
|
|
"delta":" software"
|
|
},
|
|
{
|
|
"delta":" company"
|
|
}]
|
|
}
|
|
|
|
(...)
|
|
------------------------------------------------------------
|
|
// NOTCONSOLE
|