Fixes a small typo in the `ESQLSearchResponse` (formerly known as
`ESQLSearchReponse`).
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
## Summary
tldr; ES Query alert execution creates less field_caps traffic, date
fields being accessed in alert message via `fields.*` might not render.
--
This PR reduces the number of fields loaded via field caps to the
minimum required to run a query, rather than the full field list. It
adds a `createLazy` method to the Search Source Service which internally
loads fields via a DataViewLazy object and then adds them to a DataView
object. This is to minimize changes and ship code quickly - SearchSource
objects expose the DataView object they use and kibana apps may use
this. It will take time to migrate away from this since the DataView
object is used both internally and referenced externally. A key element
of this code is the ability to extract a field list from a query so a
limited (rather than complete) set of fields can be loaded.*
One side effect of loading fewer fields is that date fields available
via `fields.*` in the alert message may no longer work. Previously, all
fields were loaded including all date fields. Now, date fields are only
loaded if they're part of the query. This has been determined to be a
small corner case and an acceptable tradeoff.
Only the ES Query rule is using this new method of loading fields. While
further work is needed before wider adoption, this should prevent
significant data transfer savings via a reduction in field_caps usage.
Depends upon https://github.com/elastic/kibana/pull/183573
---
\* We don't need to load all fields to create a query, rather we need to
load all the fields where some attribute will change the output of a
query. Sometimes the translation from KQL to DSL is the same no matter
the field type (or any other attribute) and sometimes the translation is
dependent field type and other attributes. Generally speaking, we need
the latter.
There are additional complexities - we need to know which fields are
dates (and date nanos) when their values are displayed so their values
can be made uniform. In some circumstances we need to load a set of
fields due to source field exclusion - its not supported in ES so Kibana
submits a list of individual field names.
Finally, there are times where we solve a simpler problem rather than
the problem definition. Its easier to get a list of all fields
referenced in a KQL statement instead of only getting the subset we
need. A couple of extra fields is unlikely to result in performance
degradation.
---
Places where the field list is inspected -
```
packages/kbn-es-query/src/es_query/filter_matches_index.ts
packages/kbn-es-query/src/es_query/from_nested_filter.ts
packages/kbn-es-query/src/es_query/migrate_filter.ts
packages/kbn-es-query/src/kuery/functions/exists.ts
packages/kbn-es-query/src/kuery/functions/is.ts
packages/kbn-es-query/src/kuery/functions/utils/get_fields.ts
```
This looks like its worth closer examination since it looks at the
length of the field list -
https://github.com/elastic/kibana/blob/main/packages/kbn-es-query/src/kuery/functions/is.ts#L110
Next steps -
- [x] Discuss above usage and make sure all cases are covered in this PR
- [x] Add statement to PR on lack of date formatting
- [x] Add test to verify reduction of fields requested
---------
Co-authored-by: Matthias Wilhelm <matthias.wilhelm@elastic.co>
Co-authored-by: Lukas Olson <lukas@elastic.co>
Co-authored-by: Matthias Wilhelm <ankertal@gmail.com>
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
### Summary
Removes the versioning from the _query api requests.
This ES PR removes the version from the _query requests
https://github.com/elastic/elasticsearch/pull/108919 and got backported
at 8.14 too. We need to also remove it from our side too to be in sync
with ES changes.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Removes the versioning from the _query api requests.
This ES PR removes the version from the _query requests
https://github.com/elastic/elasticsearch/pull/108919 and got backported
at 8.14 too. We need to also remove it from our side too to be in sync
with ES changes.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Revives this https://github.com/elastic/kibana/pull/181969
To do so, I had to create a new package `search-types` and move the
types I need there.
The Discovery team can take it from here.
Note: It also does a cleanup on the types I move, some of them were
declared twice.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
There has been a consistent failure in a Discover-related test set in
the kibana-ES-serverless verification job, meaning that ES-Kibana
compatibility has drifted.
Error details:
```
+ "Encountered an unknown error: status_exception
+ Root causes:
+ status_exception: Parameter validation failed for [/_search]: The http parameter [max_concurrent_shard_requests] (with value [5]) is not permitted when running in serverless mode"
+ "Encountered an error with the number of CSV rows generated fromthe search: expected rows were indeterminable, received 0."
at Context.<anonymous> (reporting.ts:182:33)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at Object.apply (wrap_function.js:73:16)
```
This tracked back to a feature added for reporting awhile back, which
created a config schema field for the `max_concurrent_shard_requests`
parameter in the search queries:
https://github.com/elastic/kibana/pull/170344/files#diff-7bceb37eef3761e1161cf04f41668dd9195bfac9fea36e734a230b5ed878a974
Most of the changes in this PR are in test code. I created "Test" which
extend protected methods in the original classes. This was done to
remove the `@ts-expect-errors` lines of code.
- Closes https://github.com/elastic/kibana/issues/100374
## Summary
This PR adds `field.customLabel` to CSV reports (column names) if
available.
Before:
<img width="800" alt="Screenshot 2024-04-24 at 15 16 45"
src="870c9146-dd3a-4cc6-a12f-f67fbd7196f5">
After:
<img width="800" alt="Screenshot 2024-04-24 at 15 17 04"
src="7f870899-71a2-42e7-ade4-d92acca9b96a">
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Due to https://github.com/elastic/kibana/issues/179641
ES is going to add a mandatory field `version`. (it is optional for now
but they want to make it required for the GA).
The version will change in backwards incompatible changes:
- syntax changes
- functions / commands output changes
This is the first PR which is adding the first version manually for now.
In https://github.com/elastic/kibana/issues/179641 we are exploring how
to deal with versioning in kibana. We are going to offer a solution in
8.15. For now we are adding the version hardcoded (otherwise when the
field is marked required kibana _query requests will fail.
👮♀️ If you are aware of any other application which uses the _query api
and is not updated in this PR please let me know
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Follows https://github.com/elastic/kibana/pull/174980
This PR updates the "task payload" data of the "new" CSV export type to
allow the `scroll` option as a method for paging the search when
generating the CSV report.
## Summary
Closes https://github.com/elastic/kibana-team/issues/715
This adds the `scroll` search API method as an option for CSV. To
achieve this, administrators can update `kibana.yml` with:
```
xpack.reporting.csv.scroll.strategy: scroll
```
The valid options for this setting are `scroll` and `pit`. The default
is `pit`.
### Design
The strategy option is only customizable in `kibana.yml` settings. It
can not be set on a per-report basis without changing the YML file and
restarting Kibana.
1. User sends a request to the Server to generate a CSV report.
2. Server creates a payload and adds a “strategy” setting from the
system configuration to the payload.
3. The Server stores the payload in the Queue.
4. The Queuing System triggers an action with the payload.
5. The system reads the “strategy” setting from the payload.
6. The system begins to export data using a method based on the
strategy.
```
User⎯Request → Server
↓
Task payload (with strategy added)
↓
Kibana Task Manager⎯Action → CSV Generator
```
### Other changes
1. Reorganize source files in the kbn-generate-csv package.
### Checklist
Delete any items that are not applicable to this PR.
- [x] Update "Inspect search in Dev Tools" for scroll option
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
close https://github.com/elastic/kibana/issues/173390
This PR enables CSV report generation with ES|QL in Discover.
Pre this PR, there are two report types for generating CSV reports with
Discover:
-
https://github.com/elastic/kibana/blob/main/packages/kbn-reporting/export_types/csv/csv_searchsource.ts
- old deprecated report type that relies on
`SerializedSearchSourceFields`. This type is still currently used by
Discover UI and we plan to migrate away from it
https://github.com/elastic/kibana/issues/151190
-
https://github.com/elastic/kibana/blob/main/packages/kbn-reporting/export_types/csv/csv_v2.ts
- new report type that relies on Discover locator. Currently, it can
only generate reports from searches backed by saved search and this
report type was implemented as a public-facing API for simple report
generation outside Kibana.
Since we plan to migrate to the v2 report type and search source is not
needed for es|ql, this PR implements es|ql csv reporting based on the v2
report type.
Initially, I wanted to create new new report type similar to v2 just for
es|ql, but it turned out to be a lot more boilerplate code without a
significant benefit. you can see it in the PR
https://github.com/elastic/kibana/pull/174448. So I changed my mind and
the current PR adds es|ql capabilities inside the existing csv_v2
report. This is convenient as the input is the same (Discover locator),
the output is the same csv file and meta information, and telemetry is
also the same.
As of this PR, the ES|QL report is capable of:
- Using es|ql query from the locator
- Adding time range filter if present in the locator. time field is
picked from the data view (it is available in the locator, but otherwise
is not used for es|ql). Other filters are also passed if available in
the locator
- Keeps into account "columns" from the locator.
- Similar to current non-es|ql reports from discover UI, it doesn't use
saved searches but only relies on the state from the URL. This probably
will be improved in https://github.com/elastic/kibana/issues/151190 to
support both.
- Uses different CSV settings for functionality like checking formulas,
escapes, bom, max size, etc...
- Keeps regular CSV features like cancelation and giving event loop a
break (even though those are not needed for now for es|ql since the
number of results is limited)
Some notable differences compared to regular discover search / csv
reports:
- A lot simpler, as it doesn't use search source and field formats
- No pagination, less CPU heavy as esql responses are limited to 10000
results and a single request
## Summary
Initial part of https://github.com/elastic/kibana/issues/131852
This PR moves towards auto-calculating the maximum timeouts calculated
based on the timing context provided by the running task instance.
### Other changes
* Added an optional logger parameter to the `getScreenshots` function.
When a logger is provided, the logs created by the screenshot plugin
will have the contextual tags added by the calling code.
* Before
<img width="1198" alt="image"
src="f68a102e-6af2-4863-aedb-52f1e4a099d8">
* After
<img width="1200" alt="image"
src="2dd4c947-ffa6-4cb3-b8a2-22893f49ddb7">
* Fixed an unreported bug where browser timezone was not utilized in PNG
reports.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
This PR fixes a **logging** issue noticed when analyzing CSV Export
logs. There is an "info" log giving the total number of hits, which
includes the `total.relation` value in the message. This log message
incorrectly printed the relationship as `eq` even when the relation was
not given in the Elasticsearch response.
The correction to this log message will help us make sure our search
requests are configured to track the total hits of the query.
```
# Before
Total hits eq 12345.
# After
Received total hits: 12345. Accuracy: eq.
Received total hits: 12345. Accuracy: unknown.
```
### Other changes
* A new private method for logging the result metadata
* Syntax cleanup
* More tests
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
This PR refactors the export type classes into their own packages to be
then instantiated in the reporting plugin. This will reduce bloat in the
central reporting plugin.
**Main packages**
- `kbn/reporting-export-types-{png,pdf,csv}` are server packages with
export type declarations
- `kbn/reporting-export-types-{png,pdf,csv}-common` are shared common
packages with type declarations and constants
**Other changes**
- Remove `reporting.getScreenshots()`
- Remove duplicated `schema_utils.ts`
- Consolidate `JOB_STATUS` declaration as an enum
<img width="1063" alt="image"
src="bced8321-93c5-4ebd-b31e-1fd946166241">
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Timothy Sullivan <tsullivan@elastic.co>
