## Summary
Based on https://github.com/elastic/elasticsearch/pull/108421
Closes https://github.com/elastic/kibana/issues/180805
It allows the users to add an `?earliest` and `?latest` variable in the
ES|QL editor.
When we are detecting this variable, we are also sending the values to
ES.
- Earliest is the from value of the date picker
- Latest is the to value of the date picker
Usage in bucket
This enables 2 very important features:
- I can use the values of the time picker in the bucket function
- I can use the time picker for indices without `@timestamp` field
### For reviewers
- Although it seems as a big PR, the majority of the changes happen due
to the signature change of the `getESQLAdHocDataview`
- The ML changes are mostly because the ML code has a lot of repetition.
I think the code needs to be refactored to have a central point
(preferably the `getESQLResults` from the esql-utils. I will create an
issue for the ML team.
- I am not proposing this in bucket autocomplete because it doesnt work
great in general for the date histogram case. We are working on
autocomplete improvements so I am expecting this to be part of a follow
up PR.
- I want to talk to the docs team to add it in the docs.
### Follow ups
- Change the histogram to use the bucket instead of the date_trunc
(needs investigation first)
- Speak with the docs team about adding an example on our official docs
### Flaky test runner
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6521
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Julia Rechkunova <julia.rechkunova@gmail.com>
## Summary
Closes https://github.com/elastic/kibana/issues/184025
This PR enables the migration from Ace to Monaco in Dev Tools Console by
default in the main branch. All serverless projects will still have the
migration disabled by default. After 8.15 is branched, the migration
will be disabled there as well. The intended release version for this
migration is 8.16.
### Functional tests
This PR creates a copy of functional tests for Monaco Console and keeps
the tests for Ace in a separate folder. When the migration is released,
we can remove the code for Ace together with tests.
The Monaco tests are not the exact copy of the Ace tests, since some
functionality and autocomplete behaviour is slightly different in the
migrated Console. For example, the auto-closing of brackets works in
Monaco when typing something, but is not kicking in in the tests.
Flaky test runner
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This is a design pass at the editor. Functionality and layout have not
been changed.
### What has changed?
- Color settings in the `esql_theme.ts`
- Added additional rule groups for color treatment (e.g. more operators
and values)
- Replaced hardcoded style values with EUI variables
- As a result, one set of theme colors now works across light and dark
mode
- Made a few small Monaco configuration (i.e. options) changes that
affect the UI/visual output (e.g. active line styles, line height,
font... the font size is the same, but I believe these other changes
will make it feel larger/more readable)
- Re-ordered the options set to more easily find what is/is not enabled
- Shortened the height of the top bar; tightened up the padding; more
closely matches the footer design
### How does it look?
*Light mode*
*Dark mode*
*Compact*
### What about the other things we saw in the Slack screenshots...
I have some other ideas cookin', but I would like to iterate on those
concepts in Figma and get additional feedback. These entail the
edge-to-edge design, less borders, consolidated header and footer,
smaller CTA buttons, full-height field list, etc. Something like this...
<details>
<summary>Additional design considerations not in this PR</summary>
</details>
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Summary
Upgrading intl packages from v2 to v6
### Packages upgrade:
- [x] Add @formatJS packages
- [x] `react-intl` Upgraded
- [x] `intl-messageformat` Upgraded
- [x] `intl-format-cache` removed
- [x] `intl-relativeformat` removed
- [x] `intl-messageformat-parser` removed
### Todo list:
- [x] Refactor HTML tags
- [x] Refactor Upgrade tags
- [x] Refactor `kbn-i18n`
- [x] Refactor `kbn-i18n-react`
- [x] Refactor `FormattedRelative` to `FormattedRelativeTime`
- [x] Refactor polyfills
- [x] Refactor IntlShape types
- [x] Rewrite Providers
- [x] Rewrite tests using i18n
- [x] Removed current pseudolocale implementation (tracker:
https://github.com/elastic/kibana/issues/180244)
- [x] Fix jest tests using rendered `Provider`
- [x] Remove no longer valid i18n packages documentation (tracker:
https://github.com/elastic/kibana/issues/180259)
Closes https://github.com/elastic/kibana/issues/178968
Closes https://github.com/elastic/kibana/issues/38642
## Notes to code reviewers
For team other than the core team, please review your plugins code
changes by filtering files by codeowners.
### Test Snapshot updates
Most of the changes are refactors of renamed functions and changed ICU
syntax.
The main updates are snapshot changes where `FormattedMessage` is now
memoized so snapshots capturing the html tree needed to be updated to
use `<Memo(MemoizedFormattedMessage)` instead of `<FormattedMessage`
### ICU now supports HTML tags:
before:
```
<FormattedMessage
defaultMessage="To buy a shoe, { link } and { cta }"
values={{
link: (
<a class="external_link" target="_blank" href="https://www.shoe.com/">
visit our website
</a>
),
cta: <strong class="important">eat a shoe</strong>,
}}
/>
```
after:
```
<FormattedMessage
defaultMessage="To buy a shoe, <a>visit our website</a> and <cta>eat a shoe</cta>"
values={{
a: msg => (
<a class="external_link" target="_blank" href="https://www.shoe.com/">
{msg}
</a>
),
cta: msg => <strong class="important">{msg}</strong>,
}}
/>
```
### Escape character to prevent ICU parsing changed from double slashes
to single quotes:
before: `\\{escaped\\}`
after: `'{escaped}'`
### No need for Intl Shape
the new packages under formatJS are written in typescript and come with
types support out of the box so no need to set types when using i18n.
Renamed `InjectedIntlProps` with `WrappedComponentProps`.
Removed `prop-types` and `intlShape` in favor of `IntlShape`.
### FormattedRelative has been renamed to FormattedRelativeTime and its
API has changed significantly. See
[FormattedRelativeTime](https://formatjs.io/docs/react-intl/upgrade-guide-3x#formattedrelativetime)
for more details.
### All tags specified must have corresponding values and will throw
error if it's missing
All tags are now parsed and expected to be formatted properly (all
opened tags must be closed).
To skip this check you can use the `ignoreTag: true` property
```
i18n.translate('xpack.apm.agentConfig.captureJmxMetrics.description', {
defaultMessage: 'This is not an HTML tag <JMX object name pattern>' +
ignoreTag: true,
}),
```
**When do I use ignore tags?**
If your message has HTML tags, it is preferred not to ignore the Tag to
have some string verification that the html tags you are adding are
properly formatted and closed.
If it the text between brackets is not an HTML tag and it is just a
fomat preference then using `ignoreTag` makes sense.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
## Summary
Cleanups the implementation of the metadata logic. There is no reason at
this point (and maybe not even in the future) to have this metadata list
be dynamic. This PR is removing this logic and cleans it up
Fixes https://github.com/elastic/kibana/issues/180218
## Summary
This PR fixes the highlighting issue in the Monaco editor that is caused
when using a theme in the output panel that is different from the theme
in the editor. This fix is implemented by unifying the themes for both
editors into one.
The PR also fixes the issue with Console theme overwriting theme in
other code editors (e.g. embeddable Console in the index detail tabs) by
reusing the `euiTheme`.
Note: This solution changes some of the original highlighting colors in
Console (e.g. comments were green and now they are grey, background
color is also different now). We might be able to fix the text colors by
specifying unique token names in the Console language (e.g. using
`consoleComment` instead of `comment` token name). cc: @yuliacech
d77d4ea3-61b5-43fa-81ef-7e4ac239aadd
1dab2eca-08e5-421b-acd4-38e0b3d91a3e
<img width="1494" alt="Screenshot 2024-04-22 at 15 34 45"
src="a4241574-77ad-42db-8707-ae38761d52d0">
<!--
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
-->
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/180959
Retrieves the indices from ast parsing. This ensures that the index
patterns we get from the `from` command is always the correct one. I
have replaced it everywhere expect from specific places where I still
use the deprecated function. I am not sure how to test the app and I
don't want to cause regressions so I prefer the responsible teams to do
the migration.
Before
Could not retrieve the index correctly
<img width="1677" alt="image"
src="77cdac00-ffff-4b91-88ba-0fc523c5f54d">
After
Correct retrieval of the index and the @timestamp info
<img width="1067" alt="image"
src="bc14718a-30f5-4f3c-8a56-cf57f69cff14">
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: dej611 <dej611@gmail.com>
## Summary
Closes https://github.com/elastic/kibana/issues/178990
This PR adds the actions buttons to the monaco editor and the
functionality for the buttons to send a request and copy the request as
a curl command. When the cursor or user selection doesn't overlap or
contain any requests on the selected line, the actions buttons are
hidden. When the cursor or selection changes, the buttons are displayed
on the 1st line of the 1st selected request. Several requests can be
sent at once. Only the 1st request is copied as a curl command.
There are also some minor UI and copy changes (see screenshots below) as
suggested by @MichaelMarcialis in the review.
### Screenshot
#### Before
<img width="882" alt="Screenshot 2024-04-15 at 15 16 59"
src="8fa95a5b-51c6-4220-8837-38adc4696602">
<img width="296" alt="Screenshot 2024-04-15 at 15 17 07"
src="6c99a4db-f7a6-4872-a24a-cddb56e0ec3d">
#### After
<img width="916" alt="Screenshot 2024-04-15 at 15 18 08"
src="1ae0b161-c731-42e1-90af-69468e9f0905">
<img width="367" alt="Screenshot 2024-04-15 at 15 18 14"
src="bb8818bd-38d4-4ec3-8bf4-30e24afd3664">
### How to test
- Check that the actions buttons are not displayed when there are no
requests in the input
- Check that the actions buttons are not displayed when no requests are
selected by the cursor or the selection
- Check that the selected requests are highlighted
- Check that the buttons are displayed on the 1st line of the 1st
selected request
- Check that the position of the buttons is updated when the editor is
scrolled up or down
### Follow up issues
- The functionality for the button to open a documentation for the 1st
selected request
- The functionality for the button to auto-indent input
- The input highlighting is temporarily reset when sending a request
- Sent requests need to be saved to history
- After sending a request, the autocomplete polling needs to be
restarted
- Add more unit tests
## Summary
The colors are not exported from eui so they are hardcoded now. I asked
from the team to expose them. I will create a follow up PR as soon as
they are exported
**Light mode**
<img width="1675" alt="image"
src="60dfd901-fa42-4ea5-b788-52836de0d837">
**Dark mode**
<img width="1679" alt="image"
src="8ea12e9c-4588-4894-8025-50bd6e7b6a48">
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Closes https://github.com/elastic/kibana/issues/178941
## Summary
This PR migrates the output editor from Ace editor to Monaco editor.
<img width="1491" alt="Screenshot 2024-04-03 at 11 22 27"
src="fbff226d-8f6b-470a-b6c8-34f4aaf64200">
**How to test:**
1. Create a `config/kibana.dev.yml` file (if one doesn't exist already)
and add the line: `console.dev.enableMonaco: true`
2. Change the condition in
b8c13babf1/src/plugins/console/public/application/containers/editor/editor.tsx (L76)
to `!isMonacoEnabled` to display the old Ace editor in the request panel
for now so that we can test sending a request.
3. Send a request and verify the output panel displays the response
correctly in json format and that the highlighting is the same as in the
original output panel. Also test sending multiple requests at once.
<details>
<summary>JSON output screenshots:</summary>
Ace editor:
<img width="561" alt="Screenshot 2024-03-22 at 16 12 48"
src="898a3e48-5a3b-48b6-ac4e-92fa6e05403a">
Now:
<img width="722" alt="Screenshot 2024-04-03 at 11 12 59"
src="503614db-946b-490e-9447-fd3cdcb83bcf">
When multiple requests are sent:
<img width="1491" alt="Screenshot 2024-04-03 at 11 10 53"
src="63b05227-a723-49ec-a9ac-3ae3269effd1">
</details>
4. Send a request with `?format=yaml` parameter (e.g. `GET
_all?format=yaml`) and verify that the output panel displays the
response correctly in yaml format.
<details>
<summary>YAML output screenshot:</summary>
<img width="1459" alt="Screenshot 2024-03-27 at 11 29 45"
src="c1967f0f-021e-4b68-bb8f-3098ef35ff1f">
</details>
5. The output also supports text output data. To test this format,
delete line 29 and add the following code in line 34 in
`src/plugins/console/public/application/containers/editor/monaco/monaco_editor_output.tsx`:
```
const data = [
{
response: { value: 'Hello World!' },
},
];
```
Then reload the Console page and verify the data is correctly displayed
and highlighted.
<details>
<summary>TEXT output screenshot:</summary>
<img width="285" alt="Screenshot 2024-03-22 at 18 06 31"
src="f30cdfd6-be67-4d34-b54d-2a19d6dcb09d">
</details>
<!---
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
-->
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR adds a theme for the Console language in Monaco editor and adds
more lexer rules to bring the highlighting of the input closed to the
original in Ace editor.
### Screenshots
Monaco editor
<img width="682" alt="Screenshot 2024-03-19 at 12 38 07"
src="98a1acc7-3a8a-4ad9-a79e-5236091c4c39">
Ace editor
<img width="651" alt="Screenshot 2024-03-19 at 12 37 52"
src="37935a68-923b-493c-ac56-ef4982f27fdf">
### How to test
1. Add `console.dev.enableMonaco: true` to `kibana.dev.yml``
2. Type different requests into Console and check that the highlighting
works the same as in Ace. For example, use the following requests
```
GET ${pathVariable}/_search
{
"query": {
"match": {
"${bodyNameVariable}": "${bodyValueVariable}",
"number_property": 1234,
"array_property": ["test1", 1234, false],
"boolean_property": true,
"text_property": "text_value",
"triple_quote": """
inside triple quote
"""
// line comment
/*
block comment
*/
}
}
}
// line comment
/*
block comment
*/
GET _sql
{
"query": """
SELECT "field" FROM "index-*" WHERE "column" = "value"
"""
}
```
3. To check that `xjson` highlighting still works
a. Navigate to Ingest pipelines and click the "create from csv" button
b. Load a valid csv file, for example this
[one](https://github.com/kgeller/ecs-mapper/blob/master/example/mapping.csv)
#### Known issues that will be addressed in follow up PRs
- SQL highlighting needs to be re-implemented (added to the follow up
list in https://github.com/elastic/kibana/issues/176926)
- Strings inside triple quotes are not using italics (added to the
follow up list in https://github.com/elastic/kibana/issues/176926)
- Font size needs to be set via settings and the default value provided
(fixed via https://github.com/elastic/kibana/pull/178982)
- Font family: do we want to use the same font as for other Monaco
languages are use the one for Ace? (added to the follow up list in
https://github.com/elastic/kibana/issues/176926)
- In the future, we might want to use the same theme for `xjson` and
Console (added to the follow up list in
https://github.com/elastic/kibana/issues/176926)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This PR updates the ES|QL grammars (lexer and parser) to match the
latest version in Elasticsearch.
## Edit by @dej611
* Introduced new `META` command (with the only `FUNCTIONS` option) who
replaced the previous `SHOW FUNCTIONS`
* add new `meta` command into the commands definitions
* make it work with both validation and autocomplete
* updated ES|QL theme with new `meta` command
* Removed the `FUNCTIONS` function from the `SHOW` command
* tweaked the `show` command definition
* fix both validation and autocomplete
* removed some unused code after the grammar changes
---------
Co-authored-by: dej611 <dej611@gmail.com>
Co-authored-by: Marco Liberati <dej611@users.noreply.github.com>
## Summary
Introduces a CI job to check for changes to the Elasticsearch grammar.
Part of https://github.com/elastic/kibana/issues/178262
The first time this job runs, it will result in a PR to update the
grammar because of formatting differences. That should be merged. Then,
it will only create a PR when something has changed on the Elasticsearch
side.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Part of https://github.com/elastic/kibana/issues/177699
We had `case` marked as if it required three parameters when in reality
it only requires two.
<img width="600" alt="Screenshot 2024-03-19 at 4 23 29 PM"
src="45f7578a-e6ad-4ba9-b71a-05bb1978a384">
Note: we could consider testing these n-1 cases to prevent this kind of
bug in the future.
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Marta Bondyra <4283304+mbondyra@users.noreply.github.com>
## Summary
Improves the wildcard validation logic to handle multiple occurrencies
of wildcards.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
Up till now, we had to define our own lexer rules for our client-side
ES|QL validation. This was because we were using an unofficial ANTLR
package (before the official ANTLR had typescript support).
Now that we are using the official ANTLR library (as of
https://github.com/elastic/kibana/pull/177211), we no longer have to
encode case insensitivity into the lexer rules themselves because the
[`caseInsensitive` option](https://github.com/antlr/antlr4/pull/3399) is
now available to us.
This means we can adopt the very [same
definitions](343b1ae1ba/x-pack/plugin/esql/src/main/antlr/EsqlBaseLexer.g4)
that Elasticsearch uses as long as we set `caseInsensitive`
(Elasticsearch handles case insensitivity at runtime).
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
Related meta [issue](https://github.com/elastic/kibana/issues/176926)
This PR adds the parser used by Ace in Console to the Console language
definition in `kbn/monaco`.
Changes introduced by this PR:
- Copy the code for `'sense_editor/mode/worker_parser'` from the file
`src/plugins/console/public/application/models/legacy_core_editor/mode/worker/worker.js`
into the `kbn/monaco` package
- Move the code for the webworker from the `xjson` folder in
`kbn/monaco` to a shared folder `ace_migration`
- Register the parser worker for the Console language in `kbn/monaco`
### How to test
#### Test the parser in Console
1. Add `console.dev.enableMonaco: true` to kibana.dev.yml
2. Open Dev Tools Console and try to type a valid request, check that
there are no red markers in the editor
3. Type an invalid request and check that there are red markers in the
editor
#### Test that `xjson` language parser still works
1. Navigate to Ingest pipelines and click the "create from csv" button
2. Load a valid csv file, for example this
[one](https://github.com/kgeller/ecs-mapper/blob/master/example/mapping.csv)
3. In the editor that now display a valid json, try changing the value
and check that red markers appear for invalid json
### Screenshots
#### Invalid request (red markers in the editor)
<img width="786" alt="Screenshot 2024-02-19 at 18 06 13"
src="bac1bdfd-c402-45f1-9b9b-a9cc29ccb123">
#### Valid request
<img width="795" alt="Screenshot 2024-02-19 at 18 06 23"
src="c06b1163-1077-43c6-bddc-1d86d0116266">
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Summary
Improve error copy based on @leemthompo suggestions.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
## Summary
Fix an issue with functions with the `minParams` configuration
(`concat`, `case`, `cidr_match`).
Validation fix:
* now validation engine understands the minimum number of args and
provide a better message based on the function signature
* if the function has a single exact signature, then make it explicit
the `exact` nature
<img width="442" alt="Screenshot 2024-02-19 at 11 46 59"
src="14aa9cb4-bce2-404b-936e-cb92ec966c2d">
<img width="227" alt="Screenshot 2024-02-19 at 11 45 18"
src="94b2a051-5cd2-4f60-b65a-c3ac77c17b85">
* if the function has some optional args in the signature, then make it
explicit that there are too few or too many args
<img width="441" alt="Screenshot 2024-02-19 at 11 48 00"
src="55acf5f7-ce6b-452d-ba49-cd38ac05120e">
<img width="443" alt="Screenshot 2024-02-19 at 11 45 03"
src="653f62d3-7ee5-44d2-91e9-aae812f08394">
* if the function has a minParams configuration, then it should make it
explicit that there are too few args:
<img width="467" alt="Screenshot 2024-02-19 at 11 41 46"
src="8a4030e0-317d-4371-abd0-11b333ad26d9">
<img width="441" alt="Screenshot 2024-02-19 at 11 41 16"
src="d8f8048e-edda-44c2-b84e-b908cd7b29a5">
<img width="446" alt="Screenshot 2024-02-19 at 11 41 04"
src="c6ac58ef-a1b7-48a6-b2ad-a8994ae2b885">
* autocomplete now understand the `minParams` and suggest the right
value in the right place
* signature hover tooltip now provides a better signature for functions
with minParams (arg should not be optional, rather mandatory until the
#minParams, optional after)
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
Sync with https://github.com/elastic/elasticsearch/pull/105611
Removes the builtin operator from validation and autocomplete.
<img width="595" alt="Screenshot 2024-02-19 at 12 04 37"
src="72f0ed8c-5bc3-400d-953b-07a833a7f413">
<img width="540" alt="Screenshot 2024-02-19 at 12 04 25"
src="5be38d83-8838-4dbe-8841-1d2d7222dc33">
Tests are currently skipped as the intention to restore it back later
on.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
This PR adds a dev config to Console to enable a migration to the Monaco
editor. The config value is `false` by default and can be changed by
adding `console.dev.enableMonaco: true` to `kibana.dev.yml`. Also a new
custom Monaco language is registered for Console in the package
`kbn/monaco` and will be worked on behind the config flag iteratively in
the next weeks.
To not let the dev config get into 8.13 release, this PR will only be
merged **after** 8.13 has been branched.
After this PR is merged, the team will start re-implementing existing
Console features in the new Monaco editor, see
https://github.com/elastic/kibana/issues/176723 and
https://github.com/elastic/kibana/issues/176926
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
The validation test suite now dumps all test cases into a JSON with a
flag that tell if the query should fail or not.
On top of that an integration test has been written, who collects the
JSON and execute it again against the ES ES|QL `_query` endpoint, then
compare the two result.
* if both results are the same (both pass or errors) then it's good to
go ✅
* if client side fails and ES pass, then it's a failure ❌
* if client side passes and ES fails, then it's bad but still ok ⚠️
* in this case the error is logged and stored into a JSON (not
committed)
All tests are ran 8 times ( stringField => `["text", "keyword"]`,
numberField => `["integer", "double", "long", "unsigned_long]` for a
totale of ~16k assertions (2000 x 8 tests).
Assertions have to be within a single test to avoid memory issues on the
CI otherwise.
~~Fixes the signature utility for functions with a minimum number of
args.~~
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
## Summary
This PR contains several fixes and sync features from the ES
counterpart, here's a short list:
* Sync with grammar changes
https://github.com/elastic/elasticsearch/pull/105325
* Added the same tests as its ES counter part.
* Sync with more grammar changes
https://github.com/elastic/elasticsearch/pull/105476
* Added the same tests as its ES bug counter part (and added more for
impacted commands)
* Revisited a couple of suggestions label texts
* Revisited some error messages in validation
* Autocomplete test suite is now testing also the final list
* some tests have changed to reflect the final ranking
* within a `stats` context it suggests functions as well
* suggestions take into account previous function used now and promote
agg functions if not used yet
* suggestion ranking is now dynamic
* pipe and comma suggestion order has been inverted (comma first, then
pipe)
* agg functions have higher ranking than everything else
* Definition updates
* `concat` now requires at least 2 args
* `auto_bucket` requires constant values as 3rd and 4th arg
* `percentile` requires a constant value as 2nd arg (optional)
* `mv_max` and `mv_min` now accept any arg type
* `st_centroid` fixed returned type
* The validation logic has been extended to handle also the constant
validation scenario.
* also, renamed indexes in the tests suite to be compliant with ES
* fixed a bug here with column validation in `row` column
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
Sync with https://github.com/elastic/elasticsearch/pull/104958 for
support of builtin fn in STATS
* validation ✅
* autocomplete ✅
* also fixed `STATS BY <field>` syntax
Sync with https://github.com/elastic/elasticsearch/pull/104913 for new
`log` function
* validation ✅ - also warning for negative values
* autocomplete ✅
Sync with https://github.com/elastic/elasticsearch/pull/105064 for
removal of `PROJECT` command
* validation ✅ (both new and legacy syntax supported)
* autocomplete ✅ (will only suggest new syntax)
Sync with https://github.com/elastic/elasticsearch/pull/105221 for
removal of mandatory brackets for `METADATA` command option
* validation ✅ (added warning deprecation message when using brackets)
* autocomplete ✅
Sync with https://github.com/elastic/elasticsearch/pull/105224 for
change of syntax for ENRICH ccq mode
* validation ✅
* autocomplete ✅ (not directly promoted, the user has to type `_` to
trigger it)
* hover ✅
* code actions ✅
Do not merge until those 5 get merged.
Additional things in this PR:
* Added more tests for `callbacks` not passed scenario
* covered more cases like those with `dissect`
* Added more tests for signature params number (calling a function with
an extra arg should return an error)
* Cleaned up some more unused code
* Improved messages on too many arguments for functions
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
More fixes for the AST walker, validation, autocomplete.
A list of areas:
* [x] Add fixes and tests for the `IS NULL` and `IS NOT NULL` which was
broken
* [x] Add fixes and tests for the `+/- value` scenario
* [x] Selecting a command from suggestions will now propose another
suggestion popup with the right list of things ™️
* [x] Add test suite to check who and when we trigger automatically a
new suggestion flow
* [x] Fix validation of no aggregation arguments in `STATS` ( `... |
STATS BY field` )
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Drew Tate <drewctate@gmail.com>
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR is a batch of fixes and (tiny) new features for validation and
autocomplete items in ES|QL client side area.
Content of the PR:
* 🐛 remove array/multi-value definition annotation for `mv_*`
functions - fix#174709
<img width="305" alt="Screenshot 2024-01-31 at 13 43 37"
src="2a458e74-9c87-4a67-8b42-0071e7a04f48">
<img width="287" alt="Screenshot 2024-01-31 at 13 43 25"
src="7205d984-d0c9-410b-8b39-c1d530c3f54f">
* while the type check has been relaxed for lists, the actual
non-multivalued argument type is retained and used for validation
<img width="758" alt="Screenshot 2024-01-31 at 13 44 06"
src="384e9dfa-aaf8-4e0f-953e-a2d656776ea7">
* 🐛 Remove array/multi-value definition annotation for `mv_expand`
command - fix#174709
<img width="556" alt="Screenshot 2024-01-31 at 14 36 19"
src="0debbccc-da9a-4e6a-bfc9-3fded61300b5">
<img width="277" alt="Screenshot 2024-01-31 at 14 36 13"
src="9438832b-d8d5-48c5-a030-b8983057546e">
* ✨ add metadata fields validation + autocomplete (fields are
retrieved via Kibana `uiSettings`) ( part of #172646 )
<img width="760" alt="Screenshot 2024-01-31 at 13 45 43"
src="b3f13d99-3e12-4586-b39c-f4ea3691a2f1">
* ✨ Quick fixes now available also for metadata fields
* 🐛 fixed autocomplete for `NOT` operation
<img width="549" alt="Screenshot 2024-01-31 at 13 46 43"
src="29b7aa7c-db66-485a-a5b0-329e66df842f">
* fixed autocomplete for `in` operation, with or without `not`:
* ✨ now a missing list bracket is correctly detected and
suggested after `in`
* ✨ the content for the list is suggested and takes into
account both the left side of the `in`/`not in` operator, but also the
current content of the list
* fixed `grok` and `dissect` in various ways
*✨ the pattern provided by autocomplete for both was not valid
in `dissect` nor useful at all, so I've changed to something more useful
like `'"%{WORD:firstWord}"'` for `grok` and `'"%{firstWord}"'` for
`dissect` to match the first word in the text field.
* 🐛 there was a bug in the validation engine as both `grok` and
`dissect` could generate new columns based on matches, so now a new
field query is fired (only when either a `GROK` or a `DISSECT` command
is detected) which enriches the set of fields available (similar to the
enrich fields)
* ✅ Added tons of tests here
* 🐛 fixed an issue with proposing an assign (`=`) operator when it
should not
* `... EVAL var0 = round( ...) <here>` not within another assignment
* `... EVAL var0 = 1 <here>` not within another assignment
* `... EVAL var0 = 1 year + 2 <here>` not within another assignment
* `... | WHERE field` should not shadow a field name in this case
* 🐛 fix an annoying auto trigger suggestion on field selection, and
kept it only for functions
* 🐛 fixed an error in console with autocomplete when a typed
function does not exists
* ✅ Add tests for the hover feature
* 🔥 Removed some unused functions detected via code coverage
analysis
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
## Summary
This PR improves the UX of a query editing experience via the quick fix
feature, typical of a full IDE.
This is a bit of an experiment and a small enhancement already, but I'm
curious on more use cases for this quick fixes option. I've addressed
some simple ones here, but let me know if you think there are scenarios
you would like to be covered.
Spellchecks fixes are computed using a levenshtein distance of 3, same
as on the ES counterpart.
Current supported features:
* index spellcheck quick fix
* wildcard support
* policy spellcheck quick fix
* field/column spellcheck quick fix
* function spellcheck quick fix
* wrong quotes quick fix for literal strings
* unquoted field/column quick fix
Testing this feature I've also found a subtle bug in the autocomplete
code on unknown functions 😅 .
This feature requires some monaco additional sub-dependencies, so bundle
size can increase.
I was looking for a `Fix all` action as well, but I suspect there's
another route for that as error markers provided to the codeAction
provider are scoped by the hovered area in the editor, so multiple fixes
are only possible for the same hovered area and in this case I'm already
handling that manually for specific cases (i.e. unquoted field triggers
2 syntax + 1 validation errors).
---------
Co-authored-by: Eyo Okon Eyo <eyo.eyo@elastic.co>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@elastic.co>
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
## Summary
This is the second part of #175288 where nested expressions are
supported in STATS scope as well.
I've tweaked the previous test suite here to check all agg functions
with nested expressions in various scenarios.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
## Summary
This PR fixes a scenario where the user types a non-existent function
into the editor:
* before a JS error was logged into the console due to a forced case
into `isFunctionArgComplete` function
* before eval functions were suggested by default in this case, now the
service just stops with an early exit.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
