This copies the first line of the description of each command to just
under the syntax so that it's "in order", before the `Parameters`
section. That way if you are reading from top to bottom you see:
```
syntax
short description
parameter names and descriptions
long description
examples
```
I've also removed the `Description` section entirely if the description
was just one sentence. So in some cases that just isn't `long
description`.
This adds some basic documentation for the `profile` option in ESQL but
doesn't really explain the results beyond "this is for human debugging."
We're not ready for any kind of specification for this thing, but it is
useful to look at.
- Added support for Booleans on Max and Min
- Added some helper methods to BitArray (`set(index, value)` and `fill(from, to, value)`). This way, the container is more similar to other BigArrays, and it's easier to work with
Part of https://github.com/elastic/elasticsearch/issues/110346, as Max
and Min are dependencies of Top.
`MAX()` currently doesn't work with doubles smaller than
`Double.MIN_VALUE` (Note that `Double.MIN_VALUE` returns the smallest
non-zero positive, not the smallest double).
This PR adds tests for Max and Min, and fixes the bug (Detected by the
tests).
Also, as the tests now generate the docs, replaced the old docs with the
generated ones, and updated the Max&Min examples.
Some work around aggregation tests, with AVG as an example:
- Added tests and autogenerated docs for AVG
- As AVG uses "complex" surrogates (A combination of functions), we can't trivially execute them without a complete plan. As I'm not sure it's worth it for most aggregations, I'm skipping those cases for now, as to avoid blocking other aggs tests.
The bad side effect of skipping those tests is that most tests in AvgTests are actually ignored (74 of 100)
These warning logs and error messages assume some level of understanding
of Elasticsearch's networking config and are not particularly
actionable. This commit adds links to the relevant section of the
manual, rewords them a little to match the terminology used in the
manual, and also documents that each node must have its own publish
address, distinct from those of all other nodes.
Currently, read_slm privilege grants access to get the ILM status,
and manage_slm grants access to start/stop ILM. This access will
be removed in the future, but needs to be deprecated before
removal. Add deprecation warning to the read_slm and manage_slm docs.
Initial support for Apache Arrow's streaming format as a response for ES|QL. It triggers based on the Accept header or the format request parameter.
Arrow has implementations in every mainstream language and is a backend of the Python Pandas library, which is extremely popular among data scientists and data analysts. Arrow's streaming format has also become the de facto standard for dataframe interchange. It is an efficient binary format that allows zero-cost deserialization by adding data access wrappers on top of memory buffers received from the network.
This PR builds on the experiment made by @nik9000 in PR #104877
Features/limitations:
- all ES|QL data types are supported
- multi-valued fields are not supported
- fields of type _source are output as JSON text in a varchar array. In a future iteration we may want to offer the choice of the more efficient CBOR and SMILE formats.
Technical details:
Arrow comes with its own memory management to handle vectors with direct memory, reference counting, etc. We don't want to use this as it conflicts with Elasticsearch's own memory management.
We therefore use the Arrow library only for the metadata objects describing the dataframe schema and the structure of the streaming format. The Arrow vector data is produced directly from ES|QL blocks.
---------
Co-authored-by: Nik Everett <nik9000@gmail.com>
This adds a `NOTE` to each comparison saying that pushing the comparison
to the search index requires that the field have an `index` and
`doc_values`. This is unique compared to the rest of Elasticsearch which
only requires an `index` and it's caused by our insistence that
comparisons only return true for single-valued fields. We can in future
accelerate comparisons without `doc_values`, but we just haven't written
that code yet.
* Create manage_seaech_connector privilege
* `manage_search_connector` -> `manage_connector` and exclude connector secrets patterns from this privilege
* Add `monitor_connector` privilege
* Update Kibana system privilege to monitor_connector for telemetry
* Rename privilege to 'manage_connector_state'
Since privilege names are often namespaced and used with globs, we want to ensure that if there's a future privilege like `manage_connector_secrets`, that it is not implicitly included in this new privileg's <name>*. By extending the privilege name to include "_state", we better namespace this distinct from any "_secrets" namespace.
* Revert "Rename privilege to 'manage_connector_state'"
This reverts commit 70b89eee76.
After further discussion with the security team, this name change is not needed after all
since the secret management privileges aren't currently prefixed with "manage_"
---------
Co-authored-by: Sean Story <sean.j.story@gmail.com>
Docs tweak with a typo fix and a clarification on how the two available
settings interact (essentially
https://github.com/elastic/elasticsearch/issues/27871). I'm also open to
including this info in the more generic settings API but feels like a
simple enough callout to add to the security API.
This PR piggy-backs on recent changes in Lucene 9.11.1
(https://github.com/apache/lucene/pull/12829,
https://github.com/apache/lucene/pull/13341/), setting the parent doc
when nested fields are present. This allows moving nested documents
along with parent ones during sorting.
With this change, sorting is now allowed on fields outside nested
objects. Sorting on fields within nested objects is still not supported
(throws an exception).
Fixes#107349
* ESQL: change from quoting from backtick to quote
For historical reasons, the source declaration inside FROM command is
treated as an identifier, using backticks (`) for escaping the value.
This is inconsistent since the source is not an identifier (field name)
but an index name which has different semantics.
`index` means a field name index while "index" means a literal with
said value.
In case of FROM, the index name/location is more like a literal (also in
unquoted form) than an identifier (that is a reference to a value).
This PR tweaks the grammar and plugs in the quoted string logic so that
both the single quote (") and triple quote (""") are allowed.
* Update grammar
* Add more tests
* Add a few more tests
* Add extra test
* Update docs/changelog/108395.yaml
* Adress review comments
* Add doc note
* Revert test rename
* Fix quoting with remote cluster
* Update docs/reference/esql/source-commands/from.asciidoc
Co-authored-by: marciw <333176+marciw@users.noreply.github.com>
---------
Co-authored-by: Bogdan Pintea <bogdan.pintea@elastic.co>
Co-authored-by: Bogdan Pintea <pintea@mailbox.org>
Co-authored-by: marciw <333176+marciw@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
