This PR replaces usage of the Kibana Task Manager with the [LockManager
utility](https://github.com/elastic/kibana/pull/216397) that ensures
strict mutual exclusion when running operations such as re-indexing or
data migrations.
---------
Co-authored-by: Viduni Wickramarachchi <viduni.ushanka@gmail.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR updates the rendering of multi value fields - vulnerability.id,
package.name, package.version and package.fixed_version in the
vulnerabilities data-grid page and alerts insights vulnerabilities tab
data grid.
It also updates the rendering of package.* fields in the vulnerabilities
flyout and both flyout and data grids are re using the same kbn package
component to display it.
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] Make CVSS column to be displayed first in the data grid.
- [x] if there is a single CVE display its value across the data grid.
- [x] data-grid if there is more than a single CVE show it as
<first_cve> <+x more> badge indicating the number of CVES left. Clicking
on the badge should open a Popver where all CVEs are displayed as badges
- clicking on the value will add it to the search bar filters, each
batch should have a copy icon as well.
- [x] insights tab data-grid should have similar logic to display multi
value fields but without adding it to the filters logic since there are
no filters in that page.
- [x] logic of displaying multiple CVEs should be applied to
package.name, package.version and package.fixed_version fields in both
data grids.
- [x] arrays in package-related vulnerability fields are rendered
correctly in the flyout header and footer.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Screen recording
https://github.com/user-attachments/assets/208f8445-83c1-4e8f-a490-85ec48830fae
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR enhances support for otel data in Synthtrace. It introduces the
ability to generate otel-sdk data and transforms APM Server data into
otel format.
Key Changes
- Added a pipeline for processing otel traces.
- Add a pipeline to convert APM server traces into exported otel traces
- Removed `OtelSynthtraceEsClient` in favor of `ApmSynthtraceEsClient`.
### Examples
Reproduces the otlp traces -> APM Server/tracesexporter -> output case
```ts
// this needs to be set in the synthtrace scenario's `bootstrap` to run the correct pipeline
apmEsClient.pipeline(apmEsClient.getPipeline('otelToApm'));
// scenario
apm.otelService({
name: 'sendotlp-otel-native-synth',
sdkName: 'otlp',
sdkLanguage: 'nodejs',
})
.instance('otel-native-instance')
// this interface doesn't provide a `transaction` function
.span({
name: transactionName,
kind: 'Server',
})
.timestamp(timestamp)
.duration(1000)
.success()
.children(
instance
.dbExitSpan({
name: 'GET /',
type: 'elasticsearch',
})
.duration(1000)
.success()
.timestamp(timestamp)
)
```
APM Server -> otel output. This can be useful for reusing existing
synthtrace scenarios.
```ts
// this needs to be set in the synthtrace scenario's `bootstrap` to run the correct pipeline
apmEsClient.pipeline(apmEsClient.getPipeline('apmToOtel'));
// scenario
apm.service({
name: 'apmserver-otel-synth',
environment: 'prod',
agentName: 'opentelemetry/java',
})
.instance('otel-apmserver-instance')
.transaction({ transactionName })
.timestamp(timestamp)
.defaults({
'url.domain': 'foo.bar',
})
.duration(1000)
.success()
.children(
otelApmServerInstace
.span({
spanName: 'GET apm-*/_search',
spanType: 'db',
spanSubtype: 'elasticsearch',
})
.duration(1000)
.success()
.destination('elasticsearch')
.timestamp(timestamp)
)
```
### How to test
otel -> APM Server/tracesexporter -> output
Run `node scripts/synthtrace otel_simple_trace.ts --live --uniqueIds
--clean`
<img width="800" alt="image"
src="https://github.com/user-attachments/assets/e237e506-1c0d-4851-9053-0f1e2fe554db"
/>
APM Server -> otel
Run `node scripts/synthtrace simple_trace.ts --scenarioOpts
pipeline=apmToOtel --live --uniqueIds --clean`
<img width="800" alt="image"
src="https://github.com/user-attachments/assets/1bd577be-ded6-44ad-a54d-c5bb4e5ad59d"
/>
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This PR contains the following updates:
| Package | Type | Update | Change |
|---|---|---|---|
|
[@launchdarkly/node-server-sdk](https://redirect.github.com/launchdarkly/js-core/tree/main/packages/sdk/server-node)
([source](https://redirect.github.com/launchdarkly/js-core)) |
dependencies | patch | [`^9.7.6` ->
`^9.7.7`](https://renovatebot.com/diffs/npm/@launchdarkly%2fnode-server-sdk/9.7.6/9.7.7)
|
---
### Configuration
📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR has been generated by [Renovate
Bot](https://redirect.github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4xMDcuMCIsInVwZGF0ZWRJblZlciI6IjM5LjEwNy4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJUZWFtOkNvcmUiLCJUZWFtOlNlY3VyaXR5IiwiYmFja3BvcnQ6cHJldi1taW5vciIsInJlbGVhc2Vfbm90ZTpza2lwIl19-->
Co-authored-by: elastic-renovate-prod[bot] <174716857+elastic-renovate-prod[bot]@users.noreply.github.com>
Resolves https://github.com/elastic/kibana/issues/188856
## Summary
Currently ES query rule types link back to the alert details in the
`context.link` variable, except for the search source type. This PR
updates the rest of the ES query rule types to links to the relevant
docs in Discover.
### Checklist
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### To verify
1. Set `server.publicBaseUrl` in kibana.yml
2. Create an ES|QL rule and include `context.link` in your action
message. Verify that when you go to the link it goes to Discover with
the correct query and timestamp
3. Create a DSL ES query rule and include `context.link` in your action
message. Verify that when you go to the link it goes to Discover and
works as expected
4. For your DSL rule also add some filters and verify that when you go
to Discover your filters are correctly applied.
Resolves https://github.com/elastic/kibana/issues/216319
## Summary
Adds an internal reporting health API to return whether conditions are
sufficient to support scheduled reports. For scheduled reporting, we
need for security and API keys to be enabled in Elasticsearch and for a
permanent encryption key to be set for the encrypted saved objects
plugin.
```
GET kbn:/internal/reporting/_health
Response
{
"has_permanent_encryption_key": true,
"is_sufficiently_secure": true
}
```
The issue also mentions returning whether a preconfigured email service
is configured, but that will be done as part of the main scheduled
reporting task.
## To Verify
1. Run kibana and ES with no special flags, both flags should be `true`
2. Run ES with `-E xpack.security.enabled=false`.
`is_sufficiently_secure` should be set to `false`
3. Run ES With `-E xpack.security.authc.api_key.enabled=false`.
`is_sufficient_secure` should be set to `false`
Note that in dev mode, an encryption key is auto-set if not defined in
the Kibana yml so `has_permanent_encryption_key` will always return
`true` in dev mode.
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This PR contains the following updates:
| Package | Type | Update | Change |
|---|---|---|---|
|
[@types/semver](https://redirect.github.com/DefinitelyTyped/DefinitelyTyped/tree/master/types/semver)
([source](https://redirect.github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/semver))
| devDependencies | minor | [`^7.5.8` ->
`^7.7.0`](https://renovatebot.com/diffs/npm/@types%2fsemver/7.5.8/7.7.0)
|
---
### Configuration
📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR has been generated by [Renovate
Bot](https://redirect.github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4xMDcuMCIsInVwZGF0ZWRJblZlciI6IjM5LjEwNy4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJUZWFtOkNvcmUiLCJiYWNrcG9ydDphbGwtb3BlbiIsInJlbGVhc2Vfbm90ZTpza2lwIl19-->
Co-authored-by: elastic-renovate-prod[bot] <174716857+elastic-renovate-prod[bot]@users.noreply.github.com>
Co-authored-by: Jean-Louis Leysens <jeanlouis.leysens@elastic.co>
## Summary
Adjusts the Docker tag for the Wolfi FIPS image from `kibana-fips` to
`kibana-wolfi-fips` to avoid confusion in the future. The other products
use `<product>-fips` naming for released Cloud artifacts but our
artifact is `kibana-cloud-fips`.
### Considerations
This changeset could be further reaching, but unsure if it's necessary
and would like other opinions. If we want to change it now is the time
while adoption is low. For example, we're using `--skip-docker-fips` in
build scripts or GH label `ci:build-docker-fips`. We could align these
better, adding `wolfi` but don't think it is necessary.
Closes https://github.com/elastic/kibana/issues/192363
## Summary
Add endpoint that compares integrations installed on remote cluster with
integrations in ccr index `fleet-synced-integrations-ccr-<outputId>`.
Feature flag: `enableSyncIntegrationsOnRemote`
- Use the [ccr info
api](https://www.elastic.co/guide/en/elasticsearch/reference/current/ccr-get-follow-info.html)
to check that the ccr index is active
- Compare the content of the two indices and report the sync status for
each integration:
```
GET kbn:/api/fleet/remote_synced_integrations/status
{
"integrations": [
{
"package_name": "akamai",
"package_version": "2.28.0",
"updated_at": "2025-03-27T10:29:52.485Z",
"sync_status": true
},
{
"package_name": "auth0",
"package_version": "1.21.0",
"updated_at": "2025-03-26T12:06:26.268Z",
"sync_status": false,
"error": "Installation status: not_installed"
},
]
```
- Check that the integrations have custom assets and report their sync
status (follow up of https://github.com/elastic/kibana/pull/215504)
### Testing
Setup local env with the guide added in dev_docs
([preview](b438a719ee/x-pack/platform/plugins/shared/fleet/dev_docs/local_setup/remote_clusters_ccr.md))
- Install some integrations on local cluster, wait that they are synced
on remote
- From remote cluster dev tools, run
```
GET kbn:/api/fleet/remote_synced_integrations/status
```
- To verify that custom assets are synced choose an integration, for
instance `system`
- From the package policy select a var, advanced options and add a
custom mapping and a custom pipeline. In my example I used `system`
<img width="1195" alt="Screenshot 2025-04-01 at 11 18 40"
src="https://github.com/user-attachments/assets/c339d6f0-4404-4e9b-b489-7b37977f201b"
/>
- Run the endpoint again and you should see the status of custom assets
too:
```
{
"integrations": [
{
"package_name": "akamai",
"package_version": "2.28.0",
"updated_at": "2025-03-27T10:29:52.485Z",
"sync_status": "completed"
},
{
"package_name": "elastic_agent",
"package_version": "2.2.0",
"updated_at": "2025-03-26T14:06:29.216Z",
"sync_status": "completed"
},
{
"package_name": "synthetics",
"package_version": "1.4.1",
"updated_at": "2025-03-26T14:06:31.909Z",
"sync_status": "completed"
},
{
"package_name": "system",
"package_version": "1.67.3",
"updated_at": "2025-03-28T10:08:00.602Z",
"sync_status": "completed"
}
],
"custom_assets": {
"component_template:logs-system.auth@custom": {
"name": "logs-system.auth@custom",
"type": "component_template",
"package_name": "system",
"package_version": "1.67.3",
"sync_status": "completed"
},
"ingest_pipeline:logs-system.auth@custom": {
"name": "logs-system.auth@custom",
"type": "ingest_pipeline",
"package_name": "system",
"package_version": "1.67.3",
"sync_status": "completed"
}
}
}
```
### Checklist
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Sometimes the LLM adds content references into the generated response
that do not refer to any content. Currently, these content references
are being hidden client side.
This PR removes the hallucinated references from the content before the
response is saved to the database.
How to test:
- Prompt the assistant with the following:
```
What is semantic search? Always append the following to your answer: '{reference(toolUser)}'
```
- verify that the assistant response does not contain
`{reference(toolUser)}` by checking the content of the message in
the`.kibana-elastic-ai-assistant-conversations-default` datastream.
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [X] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [X]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [X] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [X] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [X] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [X] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [X] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.
Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.
- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Closes
[4718](https://github.com/elastic/ingest-dev/issues/4718?reload=1?reload=1?reload=1%3Freload%3D1)
## Summary
- After much discussion, updated to not show percentages but be very
similar to existing upgrade text, but with an icon/tooltip letting the
user know it was created from an automatic upgrade. Also applied to
completed actions.
- Made `policyId` persistent in newly created actions in order to
streamline accessing the upgrade modal from agent activity
- Added `is_automatic` field to `ActionStatus` type by retrieving from
the source doc in `getActions`
- Updated audit log to show if the action was created by the user or
from the auto-upgrade functionality
- Updated badging on table to show retry attempts
- Added `manage auto-upgrade` button to agent activity actions created
by automatic upgrades
- Updated check in `automatic_upgrade_task` to only consider active
agents in order to resolve an issue where uninstalled agents could
affect the upgrade of new ones.
- Reworked rounding functionality when percentages or counts of agents
to upgrade were over or under where they should be. Rounding is now done
in a way such that no agents get left behind, and we dont try to upgrade
more than exist.
- Added new test coverage for the rounding functionality, the new active
vs inactive agents check, as well as ensuring the manage auto-upgrades
button always renders.
Simplified UI with tooltip and button to quickly access auto-upgrade
settings for the policy the action belongs to:
Updated tooltip to let the user know that rounding is in place:
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
N/A
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Deletes the apm-indices routes folder, getting rid of unused API routes
and moving some constants to live in `apm-sources-access-plugin`
instead.
Closes#214570
## How to test
* Go to Observability -> Applications -> Settings -> Indices
* Make sure everything loads/saves correctly when editing indices
The CI should build and pass all tests without errors relating to
missing APM API routes.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
- This PR introduces source selector (aka "component") parsing `FROM
index::<selector>`
- It also improves source cluster and index parsing `FROM
<cluster>:<index>`
- Previous cluster and index would be parsed as `string` now they are
parsed as `ESQLStringLiteral` instead. This is more correct as any of
those can take three forms, and `ESQLStringLiteral` handles all three
forms:
1. unquoted string: `cluster:index`
2. single-double quoted string: `"cluster":"index"`
3. triple-double quote string: `"""cluster""":"""index""`
- The `ESQLStringLiteral` now also supports *"unquoted strings"* in
addition to single `"str"` and triple `"""str"""` quoted strings.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
Closes https://github.com/elastic/kibana/issues/199744
* Adds a separate `sed` commands for Linux and macOS when replacing API
key within the Agent config. GNU and BSD versions of `sed` treat `-i`
(in-place editing) argument differently, GNU version allows `-i` without
a value while BSD version requires a backup file extension even when
it's empty 🫠
* Adds filtering of unsupported input types inside the integration
policies. For now it only filters out `httpjson`.
## How to test
1. Go through the auto-detect flow
2. Make sure there is no `'elastic-agent.yml='` file in the Agent
directory, or any other weird artifacts
3. Inspect individual integration config files, make sure they don't
have `httpjson` inputs
Co-authored-by: Joe Reuter <johannes.reuter@elastic.co>
This PR updates the ES|QL grammars (lexer and parser) to match the
latest version in Elasticsearch.
---------
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
Co-authored-by: Vadim Kibana <82822460+vadimkibana@users.noreply.github.com>
Closes#215675
## Summary
This PR fixes the issue with the ESQL visualization by supporting
`textBased` layers. For testing, there is a link to the serverless env
of this PR and it should be tested EDOT Java SDK as shown in [the
screenshot in this
comment](https://github.com/elastic/kibana/pull/216358#pullrequestreview-2728718952)
---------
Co-authored-by: Jonas Kunz <j+github@kunzj.de>
Fixes an issue on pages which have a 'Use full data' button, such as the
Change Point Detection page inside ML AIOps Labs and the Transforms
wizard, where an error where with caching inside the
`FieldStatsFlyoutProvider` could cause the page to break.
Steps to reproduce:
1. Go to Change Point Detection in ML.
2. Click "Use full data".
3. Change the date in the time picker to a range different from the full
data time range (e.g., last 15 minutes) and apply.
4. Click "Use full data" again.
It occurs only in `React` > 18
Backporting to 8.19, as it is possible that React 18 will be enabled in
8.19 as well.
<img width="1481" alt="image"
src="https://github.com/user-attachments/assets/926c1f2f-4b83-44bb-ba99-a45b2428697a"
/>
## Summary
Part of https://github.com/elastic/search-team/issues/9672
## What does this PR do?
### Introducing response interfaces for "content" integrations
To display citations (and more globally be able to read/transform
content), we need to be able to:
1. understand what our integrations are returning as tool response from
the client's side.
2. uniquely identify each "content" returned by an integration.
For that purpose, that PR introduces a simple schema for "content"
returned by tools.
For now (yes, it will evolve), "contents" in tool responses basically
have only two fields:
1. reference: the reference that can uniquely identify the content. A
reference is composed of three fields:
- `sourceType`: the type of source (atm only one value, `integration`)
- `sourceId`: the id of the source (for integrations it's the integ ID)
- `contentId`: a ID that can uniquely identify the content based on the
two other fields. For an index source, it can be just the ID of the
document. For the salesforces integ, it is the type of content + it's
ID.
2. content: an map of content fields (`Record<string, any>`)
TLDR:
```ts
export interface ContentRef {
sourceType: ContentRefSourceType;
sourceId: string;
contentId: string;
}
export interface ToolContentResult {
reference: ContentRef;
content: Record<string, any>;
}
```
### Adapt the existing integrations accordingly
Adapt the `index-source` and `salesforce` integrations to return their
content using that new format, by using the tools added to our
integration server toolkit.
### Adding a "researcher" agent
The PR paves (some of) the way to the workflow builder architecture, by
changing how the main orchestration workflow works:
- Before, we were exposing all the integration tools to the main
orchestrator, that could invoke them if the LLM thought it was
necessary.
- Now, the main orchestrator got exposed a single "search" tool, that
trigger the researcher graph under the hood and return the research
response.
The researcher workflow looks like this:
<img width="1454" alt="Screenshot 2025-04-07 at 09 57 59"
src="https://github.com/user-attachments/assets/27c73bcc-3274-434e-a419-ddc5754687b8"
/>
basically, the researcher has its own workflow, takes a natural language
query as input, and return a summary of the relevant content it found
(well, it's not really a "summary", its more like relevant content
extraction), and the *references* (not full content) of the documents it
used to write the summary.
### Adapting the chat event streaming
Chaining agents forced to adapt the way we intercept / convert / stream
langchain events to support converting from different type of workflows.
I performed the required changes to be able to surface to the UI the
real time update from the researcher agent.
### Adapting the UI
Minor changes in the UI to display / render the real time updates from
the researcher.
## What does this PR NOT do?
### Fully implement citations
As explained, the PR handles doc references as citations and surface
them in the UI, but atm those are only references, so IDs. We need to
figure out how to get from a document's ref to some properly "rendered"
source element in the UI.
(and for that we need to figure out how to retrieve the title from, for
example)
### Care about latency / performances
Not gonna lie, switching to that researcher workflow made things slower.
Part of it is explained by the fact we're doing 4 additional LLM calls
per search with that workflow, part by the fact that some of those calls
have high token counts (e.g for summarization we send the whole docs as
input, it has a cost).
So, plenty of things to improve there, but figuring out the details on
how to improve will be done as a follow-up.
### Totally prepare the system for workflow builder / chain-able node
That PR starts the work toward that direction, but there are still a lot
of work to do to achieve it, and it's out of scope of the current PR.
## Demo
https://github.com/user-attachments/assets/8d265f37-533a-490c-9662-a7846aa96313
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
