[LLM tasks] Add product documentation retrieval task (#194379)

## Summary

Close https://github.com/elastic/kibana/issues/193473
Close https://github.com/elastic/kibana/issues/193474

This PR utilize the documentation packages that are build via the tool
introduced by https://github.com/elastic/kibana/pull/193847, allowing to
install them in Kibana and expose documentation retrieval as an LLM task
that AI assistants (or other consumers) can call.

Users can now decide to install the Elastic documentation from the
assistant's config screen, which will expose a new tool for the
assistant, `retrieve_documentation` (only implemented for the o11y
assistant in the current PR, shall be done for security as a follow up).

For more information, please refer to the self-review.

## General architecture

<img width="1118" alt="Screenshot 2024-10-17 at 09 22 32"
src="https://github.com/user-attachments/assets/3df8c30a-9ccc-49ab-92ce-c204b96d6fc4">

## What this PR does

Adds two plugin:
- `productDocBase`: contains all the logic related to product
documentation installation, status, and search. This is meant to be a
"low level" components only responsible for this specific part.
- `llmTasks`: an higher level plugin that will contain various LLM tasks
to be used by assistants and genAI consumers. The intent is not to have
a single place to put all llm tasks, but more to have a default place
where we can introduce new tasks from. (fwiw, the `nlToEsql` task will
probably be moved to that plugin).

- Add a `retrieve_documentation` tool registration for the o11y
assistant
- Add a component on the o11y assistant configuration page to install
the product doc

(wiring the feature to the o11y assistant was done for testing purposes
mostly, any addition / changes / enhancement should be done by the
owning team - either in this PR or as a follow-up)

## What is NOT included in this PR:

- Wire product base feature to the security assistant (should be done by
the owning team as a follow-up)
  - installation
  - utilization as tool

- FTR tests: this is somewhat blocked by the same things we need to
figure out for https://github.com/elastic/kibana-team/issues/1271

## Screenshots 

### Installation from o11y assistant configuration page

<img width="1476" alt="Screenshot 2024-10-17 at 09 41 24"
src="https://github.com/user-attachments/assets/31daa585-9fb2-400a-a2d1-5917a262367a">

### Example of output

#### Without product documentation installed 

<img width="739" alt="Screenshot 2024-10-10 at 09 59 41"
src="https://github.com/user-attachments/assets/993fb216-6c9a-433f-bf44-f6e383d20d9d">

#### With product documentation installed

<img width="718" alt="Screenshot 2024-10-10 at 09 55 38"
src="https://github.com/user-attachments/assets/805ea4ca-8bc9-4355-a434-0ba81f8228a9">

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Alex Szabo <alex.szabo@elastic.co>
Co-authored-by: Matthias Wilhelm <matthias.wilhelm@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This commit is contained in:
Pierre Gayvallet 2024-11-19 15:28:26 +01:00 committed by GitHub
parent 8352b86f59
commit 455c781c6d
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
150 changed files with 5662 additions and 64 deletions

View file

@ -690,6 +690,10 @@ the infrastructure monitoring use-case within Kibana.
using the CURL scripts in the scripts folder.
|{kib-repo}blob/{branch}/x-pack/plugins/ai_infra/llm_tasks/README.md[llmTasks]
|This plugin contains various LLM tasks.
|{kib-repo}blob/{branch}/x-pack/plugins/observability_solution/logs_data_access/README.md[logsDataAccess]
|Exposes services to access logs data.
@ -767,6 +771,10 @@ Elastic.
|This plugin helps users learn how to use the Painless scripting language.
|{kib-repo}blob/{branch}/x-pack/plugins/ai_infra/product_doc_base/README.md[productDocBase]
|This plugin contains the product documentation base service.
|{kib-repo}blob/{branch}/x-pack/plugins/observability_solution/profiling/README.md[profiling]
|Universal Profiling provides fleet-wide, whole-system, continuous profiling with zero instrumentation. Get a comprehensive understanding of what lines of code are consuming compute resources throughout your entire fleet by visualizing your data in Kibana using the flamegraph, stacktraces, and top functions views.