mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-06-29 01:44:36 -04:00
dc24003540
This profiles additional timing information for each individual driver.
To the results from `profile` it adds the start and stop time for each
driver. That was already in the task status. To the profile and task
status it also adds the number of times the driver slept and some more
detailed history about a few of those times.
Explanation time! The compute engine splits work into some number of
`Drivers` per node. Each `Driver` is a single threaded entity - it runs
on a thread for a while then does one of three things: 1. Finishes 2.
Goes async because one of it's `Operator`s has gone async 3. Yields the
thread pool because it has run for too long
This PR measures the second two. At this point only three operators can
go async: * ENRICH * Reading from an empty exchange * Writing to a full
exchange
We're quite interested the these sleeps at the moment because they think
they may be slowing things down. Here's what it looks like when a driver
goes async because it wants to read from an empty exchange:
```
... the rest of the profile ...
"sleeps" : {
"counts" : {
"exchange empty" : 2
},
"first" : [
{
"reason" : "exchange empty",
"sleep" : "2024-08-13T19:45:57.943Z",
"sleep_millis" : 1723578357943,
"wake" : "2024-08-13T19:45:58.159Z",
"wake_millis" : 1723578358159
},
{
"reason" : "exchange empty",
"sleep" : "2024-08-13T19:45:58.164Z",
"sleep_millis" : 1723578358164,
"wake" : "2024-08-13T19:45:58.165Z",
"wake_millis" : 1723578358165
}
],
"last": [same as above]
```
Every time the driver goes async we count it in the `counts` map -
grouped by the reason the driver slept. We also record the sleep and
wake times for the first and last ten times the driver sleeps. In this
case it only slept twice, so the `first` and `last` ten times is the
same array.
This should give us a good sense about why drivers sleep while using a
limited amount of memory per driver.
|
||
|---|---|---|
| .. | ||
| appendix | ||
| description | ||
| examples | ||
| kibana | ||
| layout | ||
| parameters | ||
| signature | ||
| types | ||
| aggregation-functions.asciidoc | ||
| binary.asciidoc | ||
| cast.asciidoc | ||
| conditional-functions-and-expressions.asciidoc | ||
| date-time-functions.asciidoc | ||
| grouping-functions.asciidoc | ||
| in.asciidoc | ||
| ip-functions.asciidoc | ||
| like.asciidoc | ||
| logical.asciidoc | ||
| math-functions.asciidoc | ||
| mv-functions.asciidoc | ||
| operators.asciidoc | ||
| predicates.asciidoc | ||
| README.md | ||
| rlike.asciidoc | ||
| spatial-functions.asciidoc | ||
| string-functions.asciidoc | ||
| type-conversion-functions.asciidoc | ||
| unary.asciidoc | ||
The files in these subdirectories are generated by ESQL's test suite:
description- description of each function scraped from@FunctionInfo#descriptionexamples- examples of each function scraped from@FunctionInfo#examplesparameters- description of each function's parameters scraped from@Paramsignature- railroad diagram of the syntax to invoke each functiontypes- a table of each combination of support type for each parameter. These are generated from tests.layout- a fully generated description for each functionkibana/definition- function definitions for kibana's ESQL editorkibana/docs- the inline docs for kibana
Most functions can use the generated docs generated in the layout directory.
If we need something more custom for the function we can make a file in this
directory that can include:: any parts of the files above.
To regenerate the files for a function run its tests using gradle:
./gradlew :x-pack:plugin:esql:test -Dtests.class='*SinTests'
To regenerate the files for all functions run all of ESQL's tests using gradle:
./gradlew :x-pack:plugin:esql:test