This PR is a re-do of @charlesbp87's
https://github.com/elastic/kibana/pull/202458 so that the change can be
propagated to all required versions.
@charlesbp87 thanks for raising this. Let me know if we should propagate
the change to more versions.
---------
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
This PR:
- Updates the Kibana [Alert and action
settings](https://www.elastic.co/guide/en/kibana/current/alert-action-settings-kb.html)
page to be based off of a YAML source file
(`/docs/settings-gen/source/kibana-alert-action-settings.yml`) that is
manually converted to Asciidoc format
(`kibana-alert-action-settings.asciidoc`) by means of a Perl script
(`docs/settings-gen/parse-settings.pl`). A preview of the new, generated
page is
[here](https://kibana_bk_191787.docs-preview.app.elstc.co/guide/en/kibana/master/alert-action-settings-kb.html).
- Adds the `docs/settings-gen/parse-settings.pl` script which does the
YAML → Asciidoc conversion.
All new files are added to the `/docs/source-gen` folder.
This is a trial run updating only one page of settings in the docs.
Later, in separate PRs, we plan to convert other pages. After all Kibana
settings pages have been converted, we would ask that the Perl script be
run automatically as part of the CI whenever the YAML files in
`/docs/source-gen` are added or updated.
**Notes:**
- The Docs team is happy to own and maintain the Perl script (sorry to
use Perl - it's the only scripting language that I know).
- In time we also plan to convert all of these files from Asciidoc to
Markdown.
- When we eventually/hopefully get the rest of the Kibana settings files
converted, we will announce the settings doc process to the Kibana team
by email and/or in the Kibana newsletter.
Big thanks to the amazing @lukeelmers and @KOTungseth for guiding this!
---
Why are we doing this? We aim to:
- Create a more consistent appearance for settings across all of the
docs.
- Make it easier for people to contribute, since all Asciidoc/Markdown
formatting is handled by a script.
- Make it more apparent which settings may be missing info, such as the
default values, available options, etc.
---
P.S. I haven't worked in the Kibana repo very much and would appreciate
any help navigating the CI checks.
Rel: https://github.com/elastic/docs-projects/issues/239
## Summary
Resolves: https://github.com/elastic/kibana/issues/151463
Removes all reference to ephemeral tasks from the task manager plugin.
As well as unit and E2E tests while maintaining backwards compatibility
for `xpack.task_manager.ephemeral_tasks` flag to no-op if set. This PR
has some dependencies from the PR to remove ephemeral task support from
the alerting and actions plugin
(https://github.com/elastic/kibana/pull/197421). So it should be merged
after the other PR.
Deprecates the following configuration settings:
- xpack.task_manager.ephemeral_tasks.enabled
- xpack.task_manager.ephemeral_tasks.request_capacity
The user doesn't have to change anything on their end if they don't wish
to. This deprecation is made so if the above settings are defined,
kibana will simply do nothing.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
This PR discontinues Reporting from having dual models for determining
the privilege to generate a report, and uses Kibana feature privileges
as the single model that controls those privileges.
### Changes
1. Removes all logic that is based on following settings:
* `xpack.reporting.roles.enabled`
* `xpack.reporting.roles.allow`
The settings are still supported, but any features that use the settings
are removed.
2. Removes the detection of the settings from the Upgrade Assistant
integration
### Release note
The default system of granting users the privilege to generate reports
has changed. Rather than assigning users the `reporting_user` role,
administrators should create a custom role that grants report-creation
privileges using Kibana application privileges.
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
Correlates with https://elasticco.atlassian.net/browse/ES-9856: assign
the built-in `reporting_user` role the necessary Kibana application
privileges, and make the role not marked as deprecated.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
In this PR, I'm changing the task manager docs to mention the
`poll_interval` defaults to `500`. This will go back to 8.17 where we
change the default task claiming strategy.
cc @lcawl
## Summary
Issue: https://github.com/elastic/kibana/issues/151461
Removes all reference to ephemeral tasks in the alerting and actions
plugin. As well as unit and E2E tests while maintaining backwards
compatibility for `xpack.alerting.maxEphemeralActionsPerAlert` flag.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Follows #199033
## Summary
This PR further removes logic that uses
`xpack.reporting.csv.enablePanelActionDownload`, and removes references
of that setting from our documentation. See
https://github.com/elastic/kibana/pull/199033 for the **release note**.
### Changes
1. `enablePanelActionDownload` exists as an **optional** setting in
`packages/kbn-reporting/server/config_schema.ts`
2. The only reference to this setting marks it as unused in
`x-pack/plugins/reporting/server/config/index.ts`
3. Removes the detection of the setting from the Upgrade Assistant
integration
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
This pull request includes an update to the
`docs/settings/security-settings.asciidoc` file to clarify the default
session lifespan settings for different installation environments.
Documentation update:
*
[`docs/settings/security-settings.asciidoc`](diffhunk://#diff-97a4c4e3696b33b246f55ddd794608530b693f0a7a66ae1361a32b67c7461523L204-R204):
Clarified that the default session lifespan is 30 days for on-prem
installations and 24 hours for Elastic Cloud installations.
## Summary
Resolves https://github.com/elastic/kibana/issues/188043
This PR adds new connector which is define integration with Elastic
Inference Endpoint via [Inference
APIs](https://www.elastic.co/guide/en/elasticsearch/reference/current/inference-apis.html)
The lifecycle of the Inference Endpoint are managed by the connector
registered handlers:
- `preSaveHook` -
[create](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-inference-api.html)
new Inference Endpoint in the connector create mode (`isEdit === false`)
and
[delete](https://www.elastic.co/guide/en/elasticsearch/reference/current/delete-inference-api.html)+[create](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-inference-api.html)
in the connector edit mode (`isEdit === true`)
- `postSaveHook` - check if the connector SO was created/updated and if
not removes Inference Endpoint from preSaveHook
- `postDeleteHook` -
[delete](https://www.elastic.co/guide/en/elasticsearch/reference/current/delete-inference-api.html)
Inference Endpoint if connector was deleted.
In the Kibana Stack Management Connectors, its represented with the new
card (Technical preview badge):
<img width="1261" alt="Screenshot 2024-09-27 at 2 11 12 PM"
src="https://github.com/user-attachments/assets/dcbcce1f-06e7-4d08-8b77-0ba4105354f8">
To simplify the future integration with AI Assistants, the Connector
consists from the two main UI parts: provider selector and required
provider settings, which will be always displayed
<img width="862" alt="Screenshot 2024-10-07 at 7 59 09 AM"
src="https://github.com/user-attachments/assets/87bae493-c642-479e-b28f-6150354608dd">
and Additional options, which contains optional provider settings and
Task Type configuration:
<img width="861" alt="Screenshot 2024-10-07 at 8 00 15 AM"
src="https://github.com/user-attachments/assets/2341c034-6198-4731-8ce7-e22e6c6fb20f">
subActions corresponds to the different taskTypes Inference API
supports. Each of the task type has its own Inference Perform params.
Currently added:
- completion & completionStream
- rerank
- text_embedding
- sparse_embedding
Follow up work:
1. Collapse/expand Additional options, when the connector flyout/modal
has AI Assistant as a context (path through the extending context
implementation on the connector framework level)
2. Add support for additional params for Completion subAction to be able
to path functions
3. Add support for tokens usage Dashboard, when inference API will
include the used tokens count in the response
4. Add functionality and UX for migration from existing specific AI
connectors to the Inference connector with proper provider and
completion task
5. Integrate Connector with the AI Assistants
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
Co-authored-by: Steph Milovic <stephanie.milovic@elastic.co>
Towards https://github.com/elastic/kibana/issues/190095,
https://github.com/elastic/kibana/issues/192183,
https://github.com/elastic/kibana/issues/192185
## Summary
This PR updates the following:
- heap-to-capacity converter to take into account larger amounts of RAM,
updated this to be 16GB
- initial`maxAllowedCost` to be the default capacity of 10
- adds `xpack.alerting.maxScheduledPerMinute`,
`xpack.discovery.active_nodes_lookback`, `xpack.discovery.interval`
configs to docker
- updates the TM docs for `xpack.task_manager.capacity`
---------
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
## Summary
Part of the OTel effort. This PR adds otel datastream patterns into the
default indices that are used by the APM UI.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Caue Marcondes <caue.marcondes@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Documentation updates to distinguish the main parts of Reporting:
* service framework
* CSV export
* Screenshot export (PNG/PDF)
Additionally, this PR attempts to consistently apply the admonitions
regarding limitations of the different export types.
Part of https://github.com/elastic/kibana-team/issues/720
---------
Co-authored-by: florent-leborgne <florent.leborgne@elastic.co>
## Summary
Closes https://github.com/elastic/kibana/issues/161608
* [X] Depends on https://github.com/elastic/elasticsearch/pull/97765
* [x] Depends on https://github.com/elastic/elasticsearch/pull/107581
* [x] Add create a new report job and check the details of the templated
data stream.
* [x] Run Discover tests in Flaky Test Runner:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5999
## Release Note
Reporting internal storage has been changed from using regular indices
to a data stream configuration for a more efficient sharding strategy.
This change is not expected to have any impact to users.
## Screenshots
### Upgrade test (manual process)
Using a report generated before this change, and a report generated
after "upgrading":
Even though the two reports are in different types storage, they are
still managed by the same policy:
Looking at the details of the policy shows how the different types of
storage are used:
### Log lines
Initial startup in clean environment
```
[2024-05-13T13:22:49.138-07:00][INFO ][plugins.reporting.store] Creating ILM policy for reporting data stream: kibana-reporting
[2024-05-13T13:22:53.337-07:00][INFO ][plugins.reporting.store] Linking ILM policy to reporting data stream: .kibana-reporting, component template: kibana-reporting@custom
```
Kibana restart with ES running continuously
```
[2024-05-13T13:24:32.733-07:00][DEBUG][plugins.reporting.store] Found ILM policy kibana-reporting; skipping creation.
[2024-05-13T13:24:32.733-07:00][INFO ][plugins.reporting.store] Linking ILM policy to reporting data stream: .kibana-reporting, component template: kibana-reporting@custom
```
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
~~See
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5302
(internal link)~~
## Summary
Adds a new `xpack.security.audit.ignore_filters.users` configuration
setting. This behaves similar to the existing
`xpack.security.audit.ignore_filters.spaces` configuration setting, in
that it will filter out audit events for any of the specified users.
Resolves#183136
This PR also adds documentation for the existing
`xpack.security.audit.ignore_filters.spaces` setting, as it was
previously missing.
## Testing
1) Configure audit logging, ignoring one or more users of your choosing:
```yml
# kibana.yml
xpack.security.audit:
enabled: true
appender:
type: rolling-file
fileName: ./audit.log
layout:
type: json
ignore_filters:
- users: ["elastic"]
```
2) Start Kibana and ES with a `trial` license.
3) Login as an ignored user
4) Notice the user activity is not present in the audit logs.
5) Login as a non-ignored user
6) Notice the user activity is present in the audit logs.
## Release note:
Audit logs can be filtered by username via the
`xpack.security.audit.ignore_filters.users` configuration setting.
Closes https://github.com/elastic/kibana/issues/164104
## Summary
**Replace "Download CSV" with "Generate CSV report" to export a CSV file
from saved search panel, deprecate "Download CSV", use a config flag for
providing the deprecated feature.**
This PR uses the `xpack.reporting.csv.enablePanelActionDownload`
kibana.yml setting, which was previously unused, for choosing behavior
of CSV export in a Dashboard saved search panel, and sets the default
value to `false`. The options allow the user to download a CSV file
without creating a report (deprecated, support will be removed in the
future) or to generate a CSV report (default).
1. Use the config as a flag to switch between implementations:
- downloading a CSV file without a generated report
- generating a CSV report
2. Updated documentation
3. Refactored / cleaned up tests
4. Increased API test coverage in Serverless
5. Better error handling in
`packages/kbn-reporting/public/reporting_api_client.ts`
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
## Release Note
Kibana CSV Reporting offered a feature allowing users to download a CSV
file from a saved search panel in a dashboard, without having a report
generated. This feature is now deprecated. Now, when users need to
access saved search data from a dashboard panel as CSV, a normal report
will be generated. To access the deprecated functionality, you can add
`xpack.reporting.csv.enablePanelActionDownload: true` to kibana.yml, but
this ability will be removed in a future version of Kibana.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
~This PR still needs work (tests, mainly), so keeping it in draft for
now, but feel free to take it for a spin.~
Implements Bedrock support, specifically for the Claude models.
Architecturally, this introduces LLM adapters: one for OpenAI (which is
what we already have), and one for Bedrock/Claude. The Bedrock/Claude
adapter does the following things:
- parses data from a SerDe (an AWS concept IIUC) stream using
`@smithy/eventstream-serde-node`.
- Converts function requests and results into XML and back (to some
extent)
- some slight changes to existing functionality to achieve _some_ kind
of baseline performance with Bedrock + Claude.
Generally, GPT seems better at implicit tasks. Claude needs explicit
tasks, otherwise it will take things too literally. For instance, I had
to use a function for generating a title because Claude was too eager to
add explanations. For the `classify_esql` function, I had to add extra
instructions to stop it from requesting information that is not there.
It is prone to generating invalid XML.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Resolves https://github.com/elastic/kibana/issues/166064
## Summary
Adds the following configurations to the `kibana.yml` config:
* `xpack.actions.microsoftGraphApiScope` - overrides the default Graph
API scope value of `https://graph.microsoft.com/.default`
* `xpack.actions.microsoftExchangeUrl` - overrides the default value of
`https://login.microsoftonline.com`
This allows users in different Azure environments to customize their
endpoints as needed.
## To Verify
We are unable to test this in a different environment but we can verify
that the config overrides the defaults as expected by setting the config
values to something different and the logging out the params that are
sent to `getOAuthClientCredentialsAccessToken` in
`x-pack/plugins/stack_connectors/server/connector_types/email/send_email.ts`.
Then create an MS Exchange email connector and test it to see that the
logged values are overridden as expected.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana-team/issues/715
This adds the `scroll` search API method as an option for CSV. To
achieve this, administrators can update `kibana.yml` with:
```
xpack.reporting.csv.scroll.strategy: scroll
```
The valid options for this setting are `scroll` and `pit`. The default
is `pit`.
### Design
The strategy option is only customizable in `kibana.yml` settings. It
can not be set on a per-report basis without changing the YML file and
restarting Kibana.
1. User sends a request to the Server to generate a CSV report.
2. Server creates a payload and adds a “strategy” setting from the
system configuration to the payload.
3. The Server stores the payload in the Queue.
4. The Queuing System triggers an action with the payload.
5. The system reads the “strategy” setting from the payload.
6. The system begins to export data using a method based on the
strategy.
```
User⎯Request → Server
↓
Task payload (with strategy added)
↓
Kibana Task Manager⎯Action → CSV Generator
```
### Other changes
1. Reorganize source files in the kbn-generate-csv package.
### Checklist
Delete any items that are not applicable to this PR.
- [x] Update "Inspect search in Dev Tools" for scroll option
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
User reported that this setting has an ESS icon indicating it's
available in cloud but it's actually not available in cloud. In the
future, if we want to create a cloud PR to add this setting, the icon
can be added back at that time.
## Summary
Closes https://github.com/elastic/kibana/issues/173125
Adds a new `xpack.fleet.isAirGapped` flag to `kibana.yml` that allow
users in air-gapped environments to inform Fleet that it should "fail
fast" and skip unnecessary requests that won't be possible as Kibana
doesn't have internet access.
This PR also uses the new flag to skip the API request to the
`product_versions` API if the airgapped flag is set to true. There are
probably other places we could use this flag in a similar way, but they
can be handled separately from this PR.
### Checklist
Delete any items that are not applicable to this PR.
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
This updates the list of available Fleet settings in the Kibana docs, as
follows:
- Moves the `config` and `proxy_id` settings from the Required to the
Optional section of `xpack.fleet.outputs`. CC @nchaulet since the
settings were added to the docs via
https://github.com/elastic/kibana/pull/158771). [See orange highlight]
- Adds the `ssl` setting with the `certificate` property (this setting
is already available but doesn't appear to have been documented). [See
red highlight]
- Adds the new `secrets` setting, with SSL `key` as a property. [See
blue highlight]
- Adds an example output configuration. [See green highlight]
Rel: https://github.com/elastic/ingest-docs/issues/692
See [docs
preview](https://kibana_172637.docs-preview.app.elstc.co/guide/en/kibana/master/fleet-settings-kb.html)
---
## Summary
Ref https://github.com/elastic/kibana/issues/168629
Adds a new `keep_monitoring_alive` flag to agent policies that allows
agent environments to keep the monitoring server if they choose.
> [!note]
> `monitoring_enabled: []` needs to be explicitly set for this flag to
be honored. If `monitoring_enabled` is omitted entirely, Fleet defaults
to enabling monitoring for logs and metrics.
## To test
Create a preconfigured policy where logs/metrics monitoring is
explicitly disabled and include this new flag e.g.
```yml
xpack.fleet.agentPolicies:
- name: Test preconfigured policy
id: test-preconfigured-policy
package_policies: []
monitoring_enabled: []
keep_monitoring_alive: true
```
Then, run Kibana and verify the `monitoring` block in the full agent
policy has monitoring enabled, but logs/metrics as disabled, e.g.
You can also verify the need for `monitoring_enabled: []` by creating
another preconfigured policy e.g.
```yml
- name: Test preconfigured policy 2
id: test-preconfigured-policy-2
package_policies: []
keep_monitoring_alive: true
```
This will generate a policy e.g.
All other combinations of enabling monitoring for logs/metrics should
also generate the policies as expected. If you want to verify every
combination manually:
```yml
xpack.fleet.agentPolicies:
- name: Test preconfigured policy
id: test-preconfigured-policy
package_policies: []
monitoring_enabled: []
keep_monitoring_alive: true
- name: Test preconfigured policy 2
id: test-preconfigured-policy-2
package_policies: []
keep_monitoring_alive: true
- name: Test preconfigured policy 3
id: test-preconfigured-policy-3
package_policies: []
monitoring_enabled: ['logs']
- name: Test preconfigured policy 4
id: test-preconfigured-policy-4
package_policies: []
monitoring_enabled: ['metrics']
- name: Test preconfigured policy 5
id: test-preconfigured-policy-5
package_policies: []
monitoring_enabled: ['logs', 'metrics']
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
