## Summary
This PR contains the refactor of the Lens embeddable with the new React
architecture.
fix https://github.com/elastic/kibana/issues/174957
fixes https://github.com/elastic/kibana/issues/180672
**Current status**:
✅ Ready to review
### Notes for testing and reviewers
Other than reworking the Lens embeddable with the new architecture this
PR contains the following major changes.
#### Edit flow
The `Edit` flow has changed to in-line first using the new `Edit` API
provided by the new system
* The impact of this change can be noticed in the code on the `Canvas`
case where the Custom Lens component is instructed to avoid the inline
editing. In all the other cases in-line editing is enabled by default
now.
* Another side effect of this has been the replacement of the special
`INLINE_EDIT` action id into the regular `EDIT` action. Some tests have
been affected by this replacing the `clickEdit` function with the
`openEditorFromFlyout` one.
* The Inline editing codebase **as been reworked entirely** so make sure
to stress test this side of things.
#### Attribute service
Another important aspect changed in this PR is the `attributeService`:
this was tied to the previous Embeddable system and it is now completely
skipped. The Lens wrapper around that has been reworked to be thinner
and directly call the CM services.
* Please make sure to test thoroughly save/load SO flows
#### Transformation API (by-value <=> by-reference flow)
The new system adopts the new Transformation API (who prevents the panel
to fully reload on change).
* Please make sure to test thoroughly Visualize library <=> by value
flows
* In particular moving from one type and another should change how the
Panel Settings interpret "default" values to reset
#### Message system
Also this part of the code was partially rewritten to be more manageable
ont he embeddable surface, maintaining the core functionalities.
* Please make sure to test thoroughly error messages, warnings and info
messages
* Some scenarios to test includes
* multi-layer errors (i.e. use a broken KQL query for an
annotation/multi-layers). Check that the panel recovers correctly from
it when resolved
* Missing references
* Missing dataViews
* Wrong formatted SO
* Configuration mistakes - check that a broken config is not saveable
### Other areas to check
* Change filters in dashboard/viz and check that are correctly handled
* Check drilldowns
* Check that `Unsaved changes` are correctly detected
* Check that the panel updates correctly on `View` mode change
## Main type changes
This PR contains also some important `type` changes, here's listed:
* the `query` property now explicitly supports ES|QL query type.
* in `main` it used to work without type support
* `LensEmbeddableInput`/`LensEmbeddableOutput` types have changed, but
the type names remained the same.
## Follow ups already planned:
Some enhancements have been already collected and will be addressed in a
follow up [here](https://github.com/elastic/kibana/issues/195355)
### Tasks
<details>
<summary>Detailed list of tasks for the refactor</summary>
* New embeddable factory
* [x] Define visualization context
* [x] Define observables to track
* [x] Basic panel settings
* [x] Basic edit api
* [x] inspector api
* [x] Library services
* [x] Unified search api
* [x] Basic integrations api
* [x] State management api for inline editing
* Publish correct observables
* [x] `dataViews`
* [x] `query`
* [x] `filters`
* [x] `dataLoading`
* [x] `savedObjectId`
* Actions
* [x] View underlying data api
* Custom renderer
* [x] Basic implementation
* [x] Support callbacks
* [x] Support custom styling/paddings
* Expose
* [x] Handle searchSession
* Edit
* [x] Open panel in Lens editor
* Inline editing
* [x] rework references logic
* #180726
* integrate the logic to extract filters dataViews from filters as for
the first bug in #188545
* DSL flyout
* [x] open flyout
* [x] save
* ES|QL
* [x] open flyout on creation
* [x] open flyout on editing
* [x] save
* [x] revisit mounting logic to avoid detach if possible (not possible
yet)
* [x] explore the integration with the new `onEdit` api method used for
the inline editing~~
* [x] created panel management module and sorted it out
* [x] open in Editor
* [x] fix the save on return to dashboard
* ~~migrate by ref to by value on inline editing~~ will do it in a
follow up PR
* Add from library issues
* [x] Fix missing title and tags
* Data loading
* [x] Compute all required data params for rendering
* Render the panel
* [x] hook up user messaging system
* [x] Merge search context
* [x] Expression variables
* [x] panel settings
* [x] per panel time range
* [x] per panel filter
* test with both DSL and ES|QL mode
* Reload
* [x] on unified search updates
* [x] on config changes
* [x] on drilldown changes?
* [x] on view mode change
* Attributes service
* [x] load from library
* [x] save to library
</details>
### Pending issues:
<details>
<summary>Detailed list of issues</summary>
* [x] Unified histogram does not render in Discover
* [x] Saving to library from context menu in dashboard doesn't save the
title
* [x] When adding a vis from the library the new panel has no title
* [x] Vis disappears when opening inline editor and cancel
* Create a viz, save and return to dashboard, then edit it and cancel.
* Saving an edit inline doesn't apply the changes (i.e. changing the
chart type)
* [x] Changing the chart type on the layer panel leads to a crash
* [x] Changing the chart type won't update the visualization (via both
config panel or suggestions)
* [x] Edit a dimension will stretch the panel to overflow the fly-out
* [x] duplicating a dimension in the inline editor by drag and drop
works buggy visually
* When duplicating a panel, the new panel gets the same title rather
than “title (copy)”
* [x] by-value panels
* [x] by-reference panels
* [x] brushing throughout the timerange doesn’t work
* [x] filtering when clicking on value doesn’t work
* [x] filtering from legend doesn’t work
* [x] for lens table, the sort ascending/descending actions don’t have
an effect
* [x] filtering doesn’t display on table either
* Discover related issues
* thanks to @davismcphee investigation the source of the issue seems to
be related to the way the `abortController` is managed in the new
embeddable implementation as Discover is relying on that.
* [x] needs to investigate for a fix that restores the previous
behaviour of the `abortController` management
* [x] the hits total count is not in sync with the chart/table now
* [x] Change chart type via suggestion panel when inline editing in
Discover doesn't update the chart
* [x] Dirty panel issue (see @nickofthyme 's
[comment](https://github.com/elastic/kibana/pull/186642#discussion_r1792659477)
)
* [x] `Unsaved changes` issue (see @mbondyra
[comment](https://github.com/elastic/kibana/pull/186642#discussion_r1795384587))
* [x] Multiple errors not rendered correctly in panel when blocking
(i.e. missing field - `lens-message-list-trigger` related)
* [x] recover from a blocker error required 2 renders
* Missing SO error should not be handled for the custom render component
(legacy behaviour) but should be correctly handled for dashboard (will
be handled in a follow up PR given that is broken on `main` too)
* [x] Too many requests on Unified Histogram when in Discover (3 vs 2)
* [x] Too many request on slow queries for Unified Histogram (2 vs 1)
* [x] Annotations preview issues (chart rendering with height `0px`)
* [x] `uuid` not propagated correctly
* [x] another flavour of this was `id` not propagated correctly into the
`data-test-embeddable-id` attribute
* [x] Dispatch correctly the `render` events
* [x] refresh interval does not propagate thru the Lens custom component
in Discover (thanks to @jughosta to sort this out )
</details>
---------
Co-authored-by: Marta Bondyra <4283304+mbondyra@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Marco Vettorello <vettorello.marco@gmail.com>
Co-authored-by: Marta Bondyra <marta.bondyra@elastic.co>
Co-authored-by: Bhavya RM <bhavya@elastic.co>
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
## Summary
This PR is part of the Kibana Sustainable Architecture effort.
The goal is to start categorising Kibana packages into _generic
platform_ (`group: "platform"`) vs _solution-specific_.
```
group?: 'search' | 'security' | 'observability' | 'platform'
visibility?: 'private' | 'shared'
```
Uncategorised modules are considered to be `group: 'common', visibility:
'shared'` by default.
We want to prevent code from solution A to depend on code from solution
B.
Thus, the rules are pretty simple:
* Modules can only depend on:
* Modules in the same group
* OR modules with 'shared' visibility
* Modules in `'observability', 'security', 'search'` groups are
mandatorily `visibility: "private"`.
Long term, the goal is to re-organise packages into dedicated folders,
e.g.:
```
x-pack/platform/plugins/private
x-pack/observability/packages
```
For this first wave, we have categorised packages that seem
"straightforward":
* Any packages that have:
* at least one dependant module
* all dependants belong to the same group
* Categorise all Core packages:
* `@kbn/core-...-internal` => _platform/private_
* everything else => _platform/shared_
* Categorise as _platform/shared_ those packages that:
* Have at least one dependant in the _platform_ group.
* Don't have any `devOnly: true` dependants.
### What we ask from you, as CODEOWNERS of the _package manifests_, is
that you confirm that the categorisation is correct:
* `group: "platform", visibility: "private"` if it's a package that
should only be used from platform code, not from any solution code. It
will be loaded systematically in all serverless flavors, but solution
plugins and packages won't be able to `import` from it.
* `group: "platform", visibility: "shared"` if it's a package that can
be consumed by both platform and solutions code. It will be loaded
systematically in all serverless flavors, and anybody can import / use
code from it.
* `group: "observability" | "security" | "search", visibility:
"private"` if it's a package that is intented to be used exclusively
from a given solution. It won't be accessible nor loaded from other
solutions nor platform code.
Please refer to
[#kibana-sustainable-architecture](https://elastic.slack.com/archives/C07TCKTA22E)
for any related questions.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
- Closes https://github.com/elastic/kibana/issues/199608
## Summary
This PR changes the logic around when suggestions from lens API are
used. Previously for non-transformational query and non-time-based data
it would try to render one of lens suggestions supplying chart data via
`table` prop. Now, it would not render any chart.
Before:
- Data view mode => Static histogram configuration
- ES|QL mode and non-transformational query => _**Gets lens
suggestions.**_ If histogram chart is not possible, **_takes the first
lens suggestion for rendering the chart_**
- ES|QL mode and transformational query => Gets lens suggestions. Takes
the first lens suggestion for rendering the chart.
After:
- Data view mode => Static histogram configuration (same)
- ES|QL mode and non-transformational query => ~~_**Gets lens
suggestions.**_~~ If histogram chart is not possible, **_renders
nothing_** (updated)
- ES|QL mode and transformational query => Gets lens suggestions. Takes
the first lens suggestion for rendering the chart. (same)
### Testing
As per originally reported case:
1. `node scripts/es_archiver
--kibana-url=http://elastic:changeme@localhost:5601
--es-url=http://elastic:changeme@localhost:9200 load
test/functional/fixtures/es_archiver/many_fields`
2. Navigate to Discover, switch to ES|QL mode and enter `FROM
indices-stats | LIMIT 10`
3. No chart is expected.
Also there should be no regression for
https://github.com/elastic/kibana/pull/195863
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Davis McPhee <davis.mcphee@elastic.co>
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
Catching an invalid state of properties propagated to the UnifiedHistogram which is using the Lens embeddable in Discover, that causes a rendering error when e.g. ad hoc data views are being edited. Therefore the skipped testview can be unskipped.
## Summary
Closes#192233
Just in time for Thanksgiving - a full buffet of FIPS testing fixes
Usage of non-compliant algorithms manifest as runtime errors, so it is
imperative that we attempt to run all tests possible with Kibana in FIPS
mode. However, several overrides are needed to run Kibana in FIPS mode,
resulting in setup that make it impossible to run.
## In this PR
- Enable Unit tests for FIPS pipeline
- Enable Integration Tests for FIPS pipeline
- Enable Full FTR suite for FIPS pipeline (smoke test had originally run
a subset)
- Skip tests that break with overrides
- Fix/change tests to work in FIPS mode to maximize coverage
- Examine necessity of MD5 when installing from source (TBD based Ops PR
feed back, see self review below)
- Remove md5 from es_file_client options
## Latest Successful FIPS Test Run
https://buildkite.com/elastic/kibana-fips/builds/268
---------
Co-authored-by: Brad White <Ikuni17@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>
Co-authored-by: Larry Gregory <larry.gregory@elastic.co>
## Summary
close https://github.com/elastic/kibana/issues/194165
close https://github.com/elastic/kibana-team/issues/1245
### User-facing
<img width="1680" alt="image"
src="https://github.com/user-attachments/assets/6df4ee9f-1b4d-404c-a764-592998a1d430">
This PRs adds a new tab in the editor history component. You can star
your query from the history and then you will see it in the Starred
list. The started queries are scoped to a user and a space.
### Server
To allow starring ESQL query, this PR extends [favorites
service](https://github.com/elastic/kibana/pull/189285) with ability to
store metadata in addition to an id. To make metadata strict and in
future to support proper metadata migrations if needed, metadata needs
to be defined as schema:
```
plugins.contentManagement.favorites.registerFavoriteType('esql_query', {
typeMetadataSchema: schema.object({ query: schema.string(), timeRange:...., etc... }),
})
```
Notable changes:
- Add support for registering a favorite type and a schema for favorite
type metadata. Previosly the `dashboard` type was the only supported
type and was hardcoded
- Add `favoriteMetadata` property to a saved object mapping and make it
`enabled:false` we don't want to index it, but just want to store
metadata in addition to an id.
[code](https://github.com/elastic/kibana/pull/198362/files#diff-d1a39e36f1de11a1110520d7607e6aee7d506c76626993842cb58db012b760a2R74-R87)
- Add a 100 favorite items limit (per type per space per user). Just do
it for sanity to prevent too large objects due to metadata stored in
addtion to ids.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
Co-authored-by: Stratoula Kalafateli <stratoula1@gmail.com>
## Summary
This PR adds a fix for a regression bug introduced with the new
embeddable refactor in 8.16 .
I've added an extra 8.16 FTR test to ensure it works.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
This PR removes the `.empty` file from the `test/functional/screenshots`
directory as it's no longer needed.
### Details
The file has been introduced as part of #14122 when the directory was
cleaned. Since then, new base line screenshots have been added, so the
directory is no longer empty.
1. Speeds up functional test suite by setting the time picker via ui
settings
2. removes legacy tests from editor frame (replaced by adding some
conditions to drag_drop functional tests) as they were not the best
tests anyway
- Related to https://github.com/elastic/kibana/pull/197538
## Summary
This PR removes the support of showing stats in the field popover in
ES|QL mode as this UX will be revisited in the future to provide better
results.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Removes the code that goes through the condition if the vis comes from
new or legacy library. Pie charts or xy charts don't use vislib
implementation anymore, so the condition will always give the same
result.
## Summary
Closes https://github.com/elastic/kibana/issues/184257
This PR removes the modal warning users that unsaved changes would be
lost when navigating away from dashboard to other apps within Kibana.
The modal is not necessary since unsaved changes are saved in session
storage. The benefit is that this removes the unnecessary click for
users.
This does not impact the modal for switching to view mode from edit mode
with unsaved changes as shown below.
https://github.com/user-attachments/assets/c5bdb0ec-b040-40b0-a511-fd16ad084b90
### Checklist
- [x] Update tests
- Closes https://github.com/elastic/kibana/issues/198496
## Summary
This PR fixes an issue when the histogram request returns only a partial
result (0 or greater than 0) by adding a warning icon next to the total
hits counter and not blocking the whole page with "No results" message
(when partial result with 0 hits from histogram).
<img width="1436" alt="Screenshot 2024-10-31 at 15 45 17"
src="https://github.com/user-attachments/assets/9a769fe6-bdcf-4d20-ae6e-698a5b08d76f">
### Testing
Execute the following and open `example*` data view in Discover.
```
PUT example1
PUT example1/_mapping
{
"properties": {
"message": {
"type": "text"
},
"date": {
"type": "date"
}
}
}
PUT example1/_doc/11
{
"message": "11",
"date": "2024-11-11T12:10:30Z"
}
PUT example1/_doc/12
{
"message": "22",
"date": "2024-11-12T12:10:30Z"
}
PUT example2
PUT example2/_mapping
{
"properties": {
"message": {
"type": "keyword"
},
"date": {
"type": "date"
}
}
}
PUT example2/_doc/21
{
"message": "21",
"date": "2024-12-01T12:10:30Z"
}
PUT example2/_doc/22
{
"message": "22",
"date": "2024-12-02T12:10:30Z"
}
```
Then add `message` as a breakdown field.
Notice that the histogram gets some partial results:
<img width="1563" alt="Screenshot 2024-10-31 at 16 11 14"
src="https://github.com/user-attachments/assets/8a53f661-38a2-48f8-b082-823de77ac4f2">
Now, add a filter for `_id: 11` and notice that the histogram request
has no results (it partially failed on some shards) but Discover still
renders the table:
<img width="1564" alt="Screenshot 2024-10-31 at 16 11 31"
src="https://github.com/user-attachments/assets/e154ab5d-c5d4-4703-abd4-7bf3cd7a15fb">
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Davis McPhee <davismcphee@hotmail.com>
## Summary
Closes https://github.com/elastic/kibana/issues/184631
It keeps the chart configuration when the user is doing actions
compatible with the current query such as:
- Adding a where filter (by clicking the table, the sidebar, the chart)
- Changes the breakdown field and the field type is compatible with the
current chart
- Changing to a compatible chart type (from example from bar to line or
pie to treemap)
- Changing the query that doesnt affect the generated columns mapped to
a chart. For example adding a limit or creating a runtime field etc.
The logic depends on the suggestions. If the suggestions return the
preferred chart type, then we are going to use this. So it really
depends on the api and the type / number of columns. It is as smarter as
it can in order to not create bugs. I am quite happy with the result. It
is much better than what we have so far.
### Next steps
I would love to do the same on the dahsboard too, needs more time
though. But the changes made here will def work in favor
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
---------
Co-authored-by: Marta Bondyra <4283304+mbondyra@users.noreply.github.com>
## Summary
Hello, this PR addresses the deprecation of square brackets in FROM
METADATA declarations in Elasticsearch queries, in preparation for
Elasticsearch 9.0. Closes#196988
The changes involve removing square brackets around metadata fields
(e.g., `[metadata _id]` becomes `metadata _id`) across various parts of
the codebase. No functional changes to the UE, only internal query
syntax updates.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#_add_your_labels)
- [ ] This will appear in the **Release Notes** and follow the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
