# Backport
This will backport the following commits from `8.x` to `main`:
- [[Upgrade Assistant] Data streams reindexing
(#208083)](https://github.com/elastic/kibana/pull/208083)
<!--- Backport version: 9.6.4 -->
### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)
<!--BACKPORT [{"author":{"name":"Ahmad
Bamieh","email":"ahmad.bamyeh@elastic.co"},"sourceCommit":{"committedDate":"2025-01-30T22:33:10Z","message":"[Upgrade
Assistant] Data streams reindexing (#208083)\n\n## Data streams
reindexing\r\n\r\nThis PR implements data stream reindexing corrective
action.\r\n\r\n\r\n## Screenshots\r\nFound
here\r\nhttps://docs.google.com/document/d/1QntGX5aTbjAv-VRZoKg43McZ_obddLkIPAQp_molMvw/edit?tab=t.0#heading=h.spoobki9vra3\r\n\r\n\r\n##
Testing\r\n#### Handling kibana restarts:\r\nSince we dont maintain any
state in Kibana for the data stream\r\nreindexing we dont worry about
this scenario\r\n\r\n#### Ensuring only one node is handling the task
sync with ES:\r\nSince we dont maintain any state in Kibana for the data
stream\r\nreindexing we dont worry about this scenario\r\n\r\n#### ES
task returns an exception:\r\nThis is a new state that was introduced in
persistent tasks. We show a\r\nhuge exception banner and ask user to
retry.\r\n\r\n#### ES task fails to reindex some indices but reindexing
is complete:\r\nShow an error in the in progress page. This is an
undesired state since\r\nthe user will still be asked to reindex the
data stream again for only\r\nthose failed indices.\r\n\r\n#### ES task
fails to reindex some indices but reindexing is still
in\r\nprogress\r\nWe should a count of how many failed while showing a
progress bar.\r\n\r\n#### Pause/resume/cancel functionality\r\ndata
transform reindexing only has ‘cancel’ from ES side however
it\r\nbehaves as “pause” since it will pick up where it last was
cancelled.\r\nSince we do not maintain a state in kibana we only show a
cancel button.\r\nOnce the reindex is cancelled the user can start it
again. All the\r\ndescriptions are updated from ES with the latest
incomplete countes.\r\n\r\n#### ES task stops responding\r\nWe mark the
Kibana task as failed\r\n\r\n#### ES task returns 404\r\nBoth cancelled
tasks and never started ones return 404 from ES side.\r\nThe only way to
tell the difference is to see if the Kibana task is “in\r\nprogress”
then this means it was cancelled from ES side (canelled via\r\nAPI by
the user directly for example) and we mark it as cancelled for\r\nthe
user.\r\n\r\n\r\n#### Other main flow tests:\r\n- Tested a small data
stream reindexing job\r\n- Tested a huge data stream reindexing job\r\n-
Tested pausing a job\r\n- Tested cancelling a job\r\n- Stopping ES
during reindexing and then restarting\r\n- Reindexing in a cluster with
2 Kibana
nodes","sha":"6925d129455ccb458e08cf33ce36c08e0d5313f5","branchLabelMapping":{"^v8.16.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:prev-major","v8.18.0"],"title":"[Upgrade
Assistant] Data streams
reindexing","number":208083,"url":"https://github.com/elastic/kibana/pull/208083","mergeCommit":{"message":"[Upgrade
Assistant] Data streams reindexing (#208083)\n\n## Data streams
reindexing\r\n\r\nThis PR implements data stream reindexing corrective
action.\r\n\r\n\r\n## Screenshots\r\nFound
here\r\nhttps://docs.google.com/document/d/1QntGX5aTbjAv-VRZoKg43McZ_obddLkIPAQp_molMvw/edit?tab=t.0#heading=h.spoobki9vra3\r\n\r\n\r\n##
Testing\r\n#### Handling kibana restarts:\r\nSince we dont maintain any
state in Kibana for the data stream\r\nreindexing we dont worry about
this scenario\r\n\r\n#### Ensuring only one node is handling the task
sync with ES:\r\nSince we dont maintain any state in Kibana for the data
stream\r\nreindexing we dont worry about this scenario\r\n\r\n#### ES
task returns an exception:\r\nThis is a new state that was introduced in
persistent tasks. We show a\r\nhuge exception banner and ask user to
retry.\r\n\r\n#### ES task fails to reindex some indices but reindexing
is complete:\r\nShow an error in the in progress page. This is an
undesired state since\r\nthe user will still be asked to reindex the
data stream again for only\r\nthose failed indices.\r\n\r\n#### ES task
fails to reindex some indices but reindexing is still
in\r\nprogress\r\nWe should a count of how many failed while showing a
progress bar.\r\n\r\n#### Pause/resume/cancel functionality\r\ndata
transform reindexing only has ‘cancel’ from ES side however
it\r\nbehaves as “pause” since it will pick up where it last was
cancelled.\r\nSince we do not maintain a state in kibana we only show a
cancel button.\r\nOnce the reindex is cancelled the user can start it
again. All the\r\ndescriptions are updated from ES with the latest
incomplete countes.\r\n\r\n#### ES task stops responding\r\nWe mark the
Kibana task as failed\r\n\r\n#### ES task returns 404\r\nBoth cancelled
tasks and never started ones return 404 from ES side.\r\nThe only way to
tell the difference is to see if the Kibana task is “in\r\nprogress”
then this means it was cancelled from ES side (canelled via\r\nAPI by
the user directly for example) and we mark it as cancelled for\r\nthe
user.\r\n\r\n\r\n#### Other main flow tests:\r\n- Tested a small data
stream reindexing job\r\n- Tested a huge data stream reindexing job\r\n-
Tested pausing a job\r\n- Tested cancelling a job\r\n- Stopping ES
during reindexing and then restarting\r\n- Reindexing in a cluster with
2 Kibana
nodes","sha":"6925d129455ccb458e08cf33ce36c08e0d5313f5"}},"sourceBranch":"8.x","suggestedTargetBranches":["8.18"],"targetPullRequestStates":[{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
Some Osquery queries are expected to be long running. To accommodate
this, increase the maximum timeout in the query creation UI to 24 hours
(86400 seconds).
24 hours should allow most long-running queries, while still having a
limit that ensures misbehaving queries do not block others for an
extremely long time.
Relates to https://github.com/elastic/beats/issues/42352. Osquerybeat
will also increase its timeout limit to 24h, this change will allow the
higher timeout to be set by users in Kibana.
## Summary
This PR removes `Setup Guides` buttons.
Closes: #207026
> [!NOTE]
All the tests in
`src/platform/plugins/shared/guided_onboarding/public/components/guide_panel.test.tsx`
depend on those buttons so it was easier to remove those tests.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Users should use Lens to visualize data. Users should not use TSVB or
aggregation based panels. To this end, "easy button" UIs, such as
dashboard's "Add panel" flyout, should guide users to lens and not guide
them to legacy applications.
This PR removes `legacy` section from dashboard "Add panel" flyout.
Users can still add legacy panels to dashboards via "visualize"
application. The path is more steps, but this is as intended. Using old
stuff should be painful and require extra work as a caret to move users
to new stuff.
<img width="500" alt="Screenshot 2025-01-23 at 1 08 41 PM"
src="https://github.com/user-attachments/assets/e142cba9-a880-4ad6-a3f8-a981a03daaec"
/>
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This PR gets rid of the usage of the legacy "readdefinition" types in
the frontend except for the enrichment view, because @tonyghiani is
working on that in parallel.
I first looked into whether we can just use the existing
`IngestStreamGetResponse` type, but since it doesn't carry the name of
the stream, that is actually super annoying to use - passing the name as
a separate argument is pretty annoying and increases the risk of bugs.
Even though it's not fully "Elasticsearch"-y, I decided to add the
stream name to the `IngestStreamGetResponse` - this way the `stream`
property in the response is a regular `StreamDefinition`, so all the
functions that act on it can use it right away.
This simplifies the frontend code without hurting programmatic API
users.
This PR contains the following updates:
| Package | Update | Change |
|---|---|---|
| docker.elastic.co/wolfi/chainguard-base | digest | `d04ee51` ->
`8df3ca6` |
---
### Configuration
📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR has been generated by [Renovate
Bot](https://togithub.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40MjUuMSIsInVwZGF0ZWRJblZlciI6IjM3LjQyNS4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJUZWFtOk9wZXJhdGlvbnMiLCJiYWNrcG9ydDpza2lwIiwicmVsZWFzZV9ub3RlOnNraXAiXX0=-->
Co-authored-by: elastic-renovate-prod[bot] <174716857+elastic-renovate-prod[bot]@users.noreply.github.com>
## Summary
BUG: https://github.com/elastic/kibana/issues/208907
This PR fixes the issue where user cannot select a different connector
type after mistakenly selecting a wrong one.
### Steps to reproduce without required endpoint installation:
1. Make sure there are no connectors
2. Open "AI Assistant" on one of the security solution pages
3. Press "(+) Add connector" button
4. Connector type selection modal is visible
5. Select "Amazon Bedrock" type (or any other connector type)
6. Cancel the modal
7. Press "(+) Add connector" button again
**ISSUE**: previously selected connector type is being displayed and
there is no way to switch between types
**EXPECTED**: we should show connector type selection modal once
previous one was closed
### Issue recording
https://github.com/user-attachments/assets/48052bf1-4e00-43b7-a63e-f8a7969b9dbf
### Fixed state recording
https://github.com/user-attachments/assets/48be1cc4-0326-43a1-bd57-bb82fc1f19eb
## Details
⚠️ This PR was created by an automated tool. Please review the changes
carefully. ⚠️
Add a workflow that will comment on PRs with AsciiDoc changes.
## Why
During the migration to Elastic Docs v3, the Docs team will focus
exclusively on migrating content.
To maintain consistency, prevent conflicts, and ensure a smoother
transition we will freeze all AsciiDoc changes.
This means you will get a warning when you create AsciiDoc changes in
your PRs.
See https://github.com/elastic/docs-builder/issues/281 for details
If there are any questions, please reach out to the
@elastic/docs-engineering
---------
Co-authored-by: Brandon Morelli <brandon.morelli@elastic.co>
## Summary
We anticipate generating a single document per API version for the
foreseeable future. This PR updates our OAS document merging logic to
avoid adding the `Elastic-Api-Version` header to our spec.
This will also remove this header from current documentation.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Pre-GA, we previously modified the critical path, weekly Quality Gate
pipelines over to non-preemptible images in order to avoid the
inconsistent "agent lost" Buildkite issue that might cause unnecessary
churn.
Now that we're GA, the periodic pipelines are becoming more critical
path as well, and in addition, we're seeing a concerning increase in the
number of "agent lost" issues we're encountering, as described here:
https://github.com/elastic/security-team/issues/11700
As such, this PR is hoping to address this issue for the periodic
pipelines as well.
An invariant we have to handle somehow is if a user made additions to an
unwired data stream via the streams interface, then the underlying data
stream gets deleted.
This is "allowed", since the data stream is not managed by streams.
Currently, the UI breaks if this happens and shows error toasts when
trying to load the doc count or when trying to change processing.
This PR makes this a regular case the API can handle:
* The `GET /api/streams/<id>` endpoint does not throw, but still returns
the existing definition. A new key `data_stream_exists` indicates
whether we are in this situation
* The UI clearly communicates to the user and doesn't try to do
additional requests
<img width="838" alt="Screenshot 2025-01-24 at 16 42 23"
src="https://github.com/user-attachments/assets/92cc5a82-2bb2-4d66-b47a-057ddcff5888"
/>
Trying to update ingest via the API will still result in an error.
Another weird behavior related to that was that if a dashboard is linked
to a stale classic stream, no definition is ever saved and the stream
disappears from the list when deleted, making the dashboard link
inaccessible. This PR introduces `ensureStream` which is called by the
dashboard APIs and makes sure the definition is there if dashboard links
exist. As a side effect, this makes sure that a user can't add dashboard
links to a stream they don't have access to - IMHO we should have done
that from the start.
This does not touch wired streams - for those, the data stream getting
deleted is a breach of contract. We should still handle it gracefully,
but in this case I think we should go another route and offer a button
in the UI to use the "resync" API to reconcile the state of the streams
layer and Elasticsearch. I will look into this on a separate PR.
## Summary
Search profiler checks to see if there are indices before allowing the
execution of a search to prevent a "there are no indices" error. This PR
makes this check more performant.
In large clusters, requesting a full list of indices can be slow (up to
4 seconds) and results in excess data transfer. The indices exist api
simply returns a boolean if an index is found and remains performant on
large clusters.
Fixes https://github.com/elastic/streams-program/issues/103
This PR adds a constant keyword `stream.name` field that functions
similar to the `data_stream.dataset` field.
It will allow us to clearly associate a document or a set of documents
with their streams and filter data per stream efficiently. It's also
used to validate that documents are sent properly to the root stream
instead of targeting specific child streams directly.
The `stream.name` field reports as `keyword` in the UI, but is actually
mapped per index template as a constant keyword set to the respective
value.
## Summary
Add MapsPage to the scout core to be re-used by others.
Add rudimentary docs to show how to run these tests.
Added a `waitForRender` method.
Add test id.
---------
Co-authored-by: Robert Oskamp <traeluki@gmail.com>
Co-authored-by: Nick Partridge <nicholas.partridge@elastic.co>
Co-authored-by: Nick Partridge <nick.ryan.partridge@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
## Summary
Provides the timerange in the fetch fields api. This:
- fixes the bug where the fields are not suggested when there are system
named params such as `_tstart` and `_tend`
- makes it more performant as now it checks for fields in the selected
timerange making it more performant (especially for data in the frozen
tiers)
<img width="834" alt="image"
src="https://github.com/user-attachments/assets/ec0e6f87-3149-4a3f-a620-d7eab6a852a2"
/>
Closes: #205826Closes: #205825
**Description**
Import entities input file upload box is missing label on entity store
page.
**Preconditions**
Security -> Manage ->Entity store page
**Steps to reproduce**
1. Open Entity store page
2. Run axe-core through the page
**Changes made**
1. added required `aria-label` attributes for mentioned places
**Screen**
<img width="1223" alt="image"
src="https://github.com/user-attachments/assets/29a76c2c-76f6-4bdc-b3f4-4feefc4009eb"
/>
## Summary
Adding script to discover Scout playwright tests in Kibana repo, will be
used to build CI pipeline step (running tests per plugin in a separate
worker for the start). We can also consider using it to decide if code
change should trigger only specific plugin tests to run.
Usage:
```
node scripts/scout.js discover-playwright-configs --searchPaths x-pack/platform/plugins/private/discover_enhanced
```
Output:
```
info Searching for playwright config files in the following paths:
info - x-pack/platform/plugins/private/discover_enhanced/**/ui_tests/{playwright.config.ts,parallel.playwright.config.ts}
info
info Discovered playwright config files in '1' plugins
info [discover_enhanced] plugin:
info - x-pack/platform/plugins/private/discover_enhanced/ui_tests/parallel.playwright.config.ts
info - x-pack/platform/plugins/private/discover_enhanced/ui_tests/playwright.config.ts
```
More usage examples:
```
node scripts/scout.js discover-playwright-configs // by default will search in ['src/platform/plugins', 'x-pack/**/plugins'] and return all existing ones
node scripts/scout.js discover-playwright-configs --searchPaths x-pack/platform // platform ones under x-pack
node scripts/scout.js discover-playwright-configs --searchPaths x-pack/** // all under x-pack
```
