mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-06-29 01:44:36 -04:00
fa75c58a21
This implements the `mv_expand` command that converts multivalued fields into single valued fields by emitting one row per value and copying all other fields. --------- Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
323 lines
7 KiB
Text
323 lines
7 KiB
Text
[[esql-processing-commands]]
|
|
== ESQL processing commands
|
|
|
|
++++
|
|
<titleabbrev>Processing commands</titleabbrev>
|
|
++++
|
|
:keywords: {es}, ESQL, {es} query language, processing commands
|
|
:description: ESQL processing commands change an input table by adding, removing, or changing rows and columns.
|
|
|
|
ESQL processing commands change an input table by adding, removing, or changing
|
|
rows and columns.
|
|
|
|
image::images/esql/processing-command.svg[A processing command changing an input table,align="center"]
|
|
|
|
ESQL supports these processing commands:
|
|
|
|
* <<esql-dissect>>
|
|
* <<esql-drop>>
|
|
* <<esql-eval>>
|
|
* <<esql-grok>>
|
|
* <<esql-limit>>
|
|
* <<esql-mv_expand>>
|
|
* <<esql-project>>
|
|
* <<esql-rename>>
|
|
* <<esql-sort>>
|
|
* <<esql-stats-by>>
|
|
* <<esql-where>>
|
|
|
|
|
|
[[esql-dissect]]
|
|
=== `DISSECT`
|
|
|
|
`DISSECT` enables you to extract structured data out of a string. `DISSECT`
|
|
matches the string against a delimiter-based pattern, and extracts the specified
|
|
keys as columns.
|
|
|
|
Refer to the <<dissect-processor,dissect processor documentation>> for the
|
|
syntax of dissect patterns.
|
|
|
|
[source,esql]
|
|
----
|
|
ROW a = "1953-01-23T12:15:00Z - some text - 127.0.0.1"
|
|
| DISSECT a "%{Y}-%{M}-%{D}T%{h}:%{m}:%{s}Z - %{msg} - %{ip}"
|
|
----
|
|
|
|
[[esql-drop]]
|
|
=== `DROP`
|
|
|
|
Use `DROP` to remove columns from a table:
|
|
|
|
[source,esql]
|
|
----
|
|
FROM employees
|
|
| DROP height
|
|
----
|
|
|
|
Rather than specify each column by name, you can use wildcards to drop all
|
|
columns with a name that matches a pattern:
|
|
|
|
[source,esql]
|
|
----
|
|
FROM employees
|
|
| DROP height*
|
|
----
|
|
|
|
[[esql-eval]]
|
|
=== `EVAL`
|
|
`EVAL` enables you to add new columns to the end of a table:
|
|
|
|
[source,esql]
|
|
----
|
|
FROM employees
|
|
| PROJECT first_name, last_name, height
|
|
| EVAL height_feet = height * 3.281, height_cm = height * 100
|
|
----
|
|
|
|
If the specified column already exists, the existing column will be dropped, and
|
|
the new column will be appended to the table:
|
|
|
|
[source,esql]
|
|
----
|
|
FROM employees
|
|
| PROJECT first_name, last_name, height
|
|
| EVAL height = height * 3.281
|
|
----
|
|
|
|
[discrete]
|
|
==== Functions
|
|
`EVAL` supports various functions for calculating values. Refer to
|
|
<<esql-functions,Functions>> for more information.
|
|
|
|
[[esql-grok]]
|
|
=== `GROK`
|
|
|
|
`GROK` enables you to extract structured data out of a string. `GROK` matches
|
|
the string against patterns, based on regular expressions, and extracts the
|
|
specified patterns as columns.
|
|
|
|
Refer to the <<grok-processor,grok processor documentation>> for the syntax for
|
|
of grok patterns.
|
|
|
|
[source,esql]
|
|
----
|
|
ROW a = "12 15.5 15.6 true"
|
|
| GROK a "%{NUMBER:b:int} %{NUMBER:c:float} %{NUMBER:d:double} %{WORD:e:boolean}"
|
|
----
|
|
|
|
[[esql-limit]]
|
|
=== `LIMIT`
|
|
|
|
The `LIMIT` processing command enables you to limit the number of rows:
|
|
|
|
[source,esql]
|
|
----
|
|
FROM employees
|
|
| LIMIT 5
|
|
----
|
|
|
|
[[esql-mv_expand]]
|
|
=== `MV_EXPAND`
|
|
|
|
The `MV_EXPAND` processing command expands multivalued fields into one row per value, duplicating other fields:
|
|
|
|
[source,esql]
|
|
----
|
|
include::{esql-specs}/mv_expand.csv-spec[tag=simple]
|
|
----
|
|
|
|
[%header,format=dsv,separator=|]
|
|
|===
|
|
include::{esql-specs}/mv_expand.csv-spec[tag=simple-result]
|
|
|===
|
|
|
|
[[esql-project]]
|
|
=== `PROJECT`
|
|
|
|
The `PROJECT` command enables you to specify what columns are returned and the
|
|
order in which they are returned.
|
|
|
|
To limit the columns that are returned, use a comma-separated list of column
|
|
names. The columns are returned in the specified order:
|
|
|
|
[source,esql]
|
|
----
|
|
FROM employees
|
|
| PROJECT first_name, last_name, height
|
|
----
|
|
|
|
Rather than specify each column by name, you can use wildcards to return all
|
|
columns with a name that matches a pattern:
|
|
|
|
[source,esql]
|
|
----
|
|
FROM employees
|
|
| PROJECT h*
|
|
----
|
|
|
|
The asterisk wildcard (`*`) by itself translates to all columns that do not
|
|
match the other arguments. This query will first return all columns with a name
|
|
that starts with an h, followed by all other columns:
|
|
|
|
[source,esql]
|
|
----
|
|
FROM employees
|
|
| PROJECT h*, *
|
|
----
|
|
|
|
[[esql-rename]]
|
|
=== `RENAME`
|
|
|
|
Use `RENAME` to rename a column. If a column with the new name already exists,
|
|
it will be replaced by the new column.
|
|
|
|
[source,esql]
|
|
----
|
|
FROM employees
|
|
| PROJECT first_name, last_name, still_hired
|
|
| RENAME employed = still_hired
|
|
----
|
|
|
|
Multiple columns can be renamed with a single `RENAME` command:
|
|
|
|
[source,esql]
|
|
----
|
|
FROM employees
|
|
| PROJECT first_name, last_name
|
|
| RENAME fn = first_name, ln = last_name
|
|
----
|
|
|
|
[[esql-sort]]
|
|
=== `SORT`
|
|
Use the `SORT` command to sort rows on one or more fields:
|
|
|
|
[source,esql]
|
|
----
|
|
FROM employees
|
|
| PROJECT first_name, last_name, height
|
|
| SORT height
|
|
----
|
|
|
|
The default sort order is ascending. Set an explicit sort order using `ASC` or
|
|
`DESC`:
|
|
|
|
[source,esql]
|
|
----
|
|
FROM employees
|
|
| PROJECT first_name, last_name, height
|
|
| SORT height DESC
|
|
----
|
|
|
|
If two rows have the same sort key, the original order will be preserved. You
|
|
can provide additional sort expressions to act as tie breakers:
|
|
|
|
[source,esql]
|
|
----
|
|
FROM employees
|
|
| PROJECT first_name, last_name, height
|
|
| SORT height DESC, first_name ASC
|
|
----
|
|
|
|
[discrete]
|
|
==== `null` values
|
|
By default, `null` values are treated as being larger than any other value. With
|
|
an ascending sort order, `null` values are sorted last, and with a descending
|
|
sort order, `null` values are sorted first. You can change that by providing
|
|
`NULLS FIRST` or `NULLS LAST`:
|
|
|
|
[source,esql]
|
|
----
|
|
FROM employees
|
|
| PROJECT first_name, last_name, height
|
|
| SORT first_name ASC NULLS FIRST
|
|
----
|
|
|
|
[[esql-stats-by]]
|
|
=== `STATS ... BY`
|
|
Use `STATS ... BY` to group rows according to a common value and calculate one
|
|
or more aggregated values over the grouped rows.
|
|
|
|
[source,esql]
|
|
----
|
|
FROM employees
|
|
| STATS count = COUNT(languages) BY languages
|
|
----
|
|
|
|
If `BY` is omitted, the output table contains exactly one row with the
|
|
aggregations applied over the entire dataset:
|
|
|
|
[source,esql]
|
|
----
|
|
FROM employees
|
|
| STATS avg_lang = AVG(languages)
|
|
----
|
|
|
|
It's possible to calculate multiple values:
|
|
|
|
[source,esql]
|
|
----
|
|
FROM employees
|
|
| STATS avg_lang = AVG(languages), max_lang = MAX(languages)
|
|
----
|
|
|
|
It's also possible to group by multiple values (only supported for long and
|
|
keyword family fields):
|
|
|
|
[source,esql]
|
|
----
|
|
FROM employees
|
|
| EVAL hired = DATE_FORMAT(hire_date, "YYYY")
|
|
| STATS avg_salary = AVG(salary) BY hired, languages.long
|
|
| EVAL avg_salary = ROUND(avg_salary)
|
|
| SORT hired, languages.long
|
|
----
|
|
|
|
The following aggregation functions are supported:
|
|
|
|
* `AVG`
|
|
* `COUNT`
|
|
* `COUNT_DISTINCT`
|
|
* `MAX`
|
|
* `MEDIAN`
|
|
* `MEDIAN_ABSOLUTE_DEVIATION`
|
|
* `MIN`
|
|
* `SUM`
|
|
|
|
[[esql-where]]
|
|
=== `WHERE`
|
|
|
|
Use `WHERE` to produce a table that contains all the rows from the input table
|
|
for which the provided condition evaluates to `true`:
|
|
|
|
[source,esql]
|
|
----
|
|
FROM employees
|
|
| PROJECT first_name, last_name, still_hired
|
|
| WHERE still_hired == true
|
|
----
|
|
|
|
Which, if `still_hired` is a boolean field, can be simplified to:
|
|
|
|
[source,esql]
|
|
----
|
|
FROM employees
|
|
| PROJECT first_name, last_name, still_hired
|
|
| WHERE still_hired
|
|
----
|
|
|
|
[discrete]
|
|
==== Operators
|
|
|
|
Refer to <<esql-operators>> for an overview of the supported operators.
|
|
|
|
[discrete]
|
|
==== Functions
|
|
`WHERE` supports various functions for calculating values. Refer to
|
|
<<esql-functions,Functions>> for more information.
|
|
|
|
[source,esql]
|
|
----
|
|
FROM employees
|
|
| PROJECT first_name, last_name, height
|
|
| WHERE length(first_name) < 4
|
|
----
|