mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-06-28 17:34:17 -04:00
e78bdc953a
Initial support for Apache Arrow's streaming format as a response for ES|QL. It triggers based on the Accept header or the format request parameter. Arrow has implementations in every mainstream language and is a backend of the Python Pandas library, which is extremely popular among data scientists and data analysts. Arrow's streaming format has also become the de facto standard for dataframe interchange. It is an efficient binary format that allows zero-cost deserialization by adding data access wrappers on top of memory buffers received from the network. This PR builds on the experiment made by @nik9000 in PR #104877 Features/limitations: - all ES|QL data types are supported - multi-valued fields are not supported - fields of type _source are output as JSON text in a varchar array. In a future iteration we may want to offer the choice of the more efficient CBOR and SMILE formats. Technical details: Arrow comes with its own memory management to handle vectors with direct memory, reference counting, etc. We don't want to use this as it conflicts with Elasticsearch's own memory management. We therefore use the Arrow library only for the metadata objects describing the dataframe schema and the structure of the streaming format. The Arrow vector data is produced directly from ES|QL blocks. --------- Co-authored-by: Nik Everett <nik9000@gmail.com> |
||
|---|---|---|
| .. | ||
| functions | ||
| processing-commands | ||
| source-commands | ||
| esql-across-clusters.asciidoc | ||
| esql-apis.asciidoc | ||
| esql-async-query-api.asciidoc | ||
| esql-async-query-delete-api.asciidoc | ||
| esql-async-query-get-api.asciidoc | ||
| esql-commands.asciidoc | ||
| esql-enrich-data.asciidoc | ||
| esql-examples.asciidoc | ||
| esql-functions-operators.asciidoc | ||
| esql-get-started.asciidoc | ||
| esql-kibana.asciidoc | ||
| esql-language.asciidoc | ||
| esql-limitations.asciidoc | ||
| esql-process-data-with-dissect-grok.asciidoc | ||
| esql-query-api.asciidoc | ||
| esql-rest.asciidoc | ||
| esql-security-solution.asciidoc | ||
| esql-syntax.asciidoc | ||
| esql-using.asciidoc | ||
| implicit-casting.asciidoc | ||
| index.asciidoc | ||
| metadata-fields.asciidoc | ||
| multivalued-fields.asciidoc | ||
| task-management.asciidoc | ||