## Summary
Closes https://github.com/elastic/kibana/issues/161608
* [X] Depends on https://github.com/elastic/elasticsearch/pull/97765
* [x] Depends on https://github.com/elastic/elasticsearch/pull/107581
* [x] Add create a new report job and check the details of the templated
data stream.
* [x] Run Discover tests in Flaky Test Runner:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5999
## Release Note
Reporting internal storage has been changed from using regular indices
to a data stream configuration for a more efficient sharding strategy.
This change is not expected to have any impact to users.
## Screenshots
### Upgrade test (manual process)
Using a report generated before this change, and a report generated
after "upgrading":
Even though the two reports are in different types storage, they are
still managed by the same policy:
Looking at the details of the policy shows how the different types of
storage are used:
### Log lines
Initial startup in clean environment
```
[2024-05-13T13:22:49.138-07:00][INFO ][plugins.reporting.store] Creating ILM policy for reporting data stream: kibana-reporting
[2024-05-13T13:22:53.337-07:00][INFO ][plugins.reporting.store] Linking ILM policy to reporting data stream: .kibana-reporting, component template: kibana-reporting@custom
```
Kibana restart with ES running continuously
```
[2024-05-13T13:24:32.733-07:00][DEBUG][plugins.reporting.store] Found ILM policy kibana-reporting; skipping creation.
[2024-05-13T13:24:32.733-07:00][INFO ][plugins.reporting.store] Linking ILM policy to reporting data stream: .kibana-reporting, component template: kibana-reporting@custom
```
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
~~See
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5302
(internal link)~~
## Summary
Adds a new `xpack.security.audit.ignore_filters.users` configuration
setting. This behaves similar to the existing
`xpack.security.audit.ignore_filters.spaces` configuration setting, in
that it will filter out audit events for any of the specified users.
Resolves#183136
This PR also adds documentation for the existing
`xpack.security.audit.ignore_filters.spaces` setting, as it was
previously missing.
## Testing
1) Configure audit logging, ignoring one or more users of your choosing:
```yml
# kibana.yml
xpack.security.audit:
enabled: true
appender:
type: rolling-file
fileName: ./audit.log
layout:
type: json
ignore_filters:
- users: ["elastic"]
```
2) Start Kibana and ES with a `trial` license.
3) Login as an ignored user
4) Notice the user activity is not present in the audit logs.
5) Login as a non-ignored user
6) Notice the user activity is present in the audit logs.
## Release note:
Audit logs can be filtered by username via the
`xpack.security.audit.ignore_filters.users` configuration setting.
Closes https://github.com/elastic/kibana/issues/164104
## Summary
**Replace "Download CSV" with "Generate CSV report" to export a CSV file
from saved search panel, deprecate "Download CSV", use a config flag for
providing the deprecated feature.**
This PR uses the `xpack.reporting.csv.enablePanelActionDownload`
kibana.yml setting, which was previously unused, for choosing behavior
of CSV export in a Dashboard saved search panel, and sets the default
value to `false`. The options allow the user to download a CSV file
without creating a report (deprecated, support will be removed in the
future) or to generate a CSV report (default).
1. Use the config as a flag to switch between implementations:
- downloading a CSV file without a generated report
- generating a CSV report
2. Updated documentation
3. Refactored / cleaned up tests
4. Increased API test coverage in Serverless
5. Better error handling in
`packages/kbn-reporting/public/reporting_api_client.ts`
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
## Release Note
Kibana CSV Reporting offered a feature allowing users to download a CSV
file from a saved search panel in a dashboard, without having a report
generated. This feature is now deprecated. Now, when users need to
access saved search data from a dashboard panel as CSV, a normal report
will be generated. To access the deprecated functionality, you can add
`xpack.reporting.csv.enablePanelActionDownload: true` to kibana.yml, but
this ability will be removed in a future version of Kibana.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
~This PR still needs work (tests, mainly), so keeping it in draft for
now, but feel free to take it for a spin.~
Implements Bedrock support, specifically for the Claude models.
Architecturally, this introduces LLM adapters: one for OpenAI (which is
what we already have), and one for Bedrock/Claude. The Bedrock/Claude
adapter does the following things:
- parses data from a SerDe (an AWS concept IIUC) stream using
`@smithy/eventstream-serde-node`.
- Converts function requests and results into XML and back (to some
extent)
- some slight changes to existing functionality to achieve _some_ kind
of baseline performance with Bedrock + Claude.
Generally, GPT seems better at implicit tasks. Claude needs explicit
tasks, otherwise it will take things too literally. For instance, I had
to use a function for generating a title because Claude was too eager to
add explanations. For the `classify_esql` function, I had to add extra
instructions to stop it from requesting information that is not there.
It is prone to generating invalid XML.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Resolves https://github.com/elastic/kibana/issues/166064
## Summary
Adds the following configurations to the `kibana.yml` config:
* `xpack.actions.microsoftGraphApiScope` - overrides the default Graph
API scope value of `https://graph.microsoft.com/.default`
* `xpack.actions.microsoftExchangeUrl` - overrides the default value of
`https://login.microsoftonline.com`
This allows users in different Azure environments to customize their
endpoints as needed.
## To Verify
We are unable to test this in a different environment but we can verify
that the config overrides the defaults as expected by setting the config
values to something different and the logging out the params that are
sent to `getOAuthClientCredentialsAccessToken` in
`x-pack/plugins/stack_connectors/server/connector_types/email/send_email.ts`.
Then create an MS Exchange email connector and test it to see that the
logged values are overridden as expected.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana-team/issues/715
This adds the `scroll` search API method as an option for CSV. To
achieve this, administrators can update `kibana.yml` with:
```
xpack.reporting.csv.scroll.strategy: scroll
```
The valid options for this setting are `scroll` and `pit`. The default
is `pit`.
### Design
The strategy option is only customizable in `kibana.yml` settings. It
can not be set on a per-report basis without changing the YML file and
restarting Kibana.
1. User sends a request to the Server to generate a CSV report.
2. Server creates a payload and adds a “strategy” setting from the
system configuration to the payload.
3. The Server stores the payload in the Queue.
4. The Queuing System triggers an action with the payload.
5. The system reads the “strategy” setting from the payload.
6. The system begins to export data using a method based on the
strategy.
```
User⎯Request → Server
↓
Task payload (with strategy added)
↓
Kibana Task Manager⎯Action → CSV Generator
```
### Other changes
1. Reorganize source files in the kbn-generate-csv package.
### Checklist
Delete any items that are not applicable to this PR.
- [x] Update "Inspect search in Dev Tools" for scroll option
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
User reported that this setting has an ESS icon indicating it's
available in cloud but it's actually not available in cloud. In the
future, if we want to create a cloud PR to add this setting, the icon
can be added back at that time.
## Summary
Closes https://github.com/elastic/kibana/issues/173125
Adds a new `xpack.fleet.isAirGapped` flag to `kibana.yml` that allow
users in air-gapped environments to inform Fleet that it should "fail
fast" and skip unnecessary requests that won't be possible as Kibana
doesn't have internet access.
This PR also uses the new flag to skip the API request to the
`product_versions` API if the airgapped flag is set to true. There are
probably other places we could use this flag in a similar way, but they
can be handled separately from this PR.
### Checklist
Delete any items that are not applicable to this PR.
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
This updates the list of available Fleet settings in the Kibana docs, as
follows:
- Moves the `config` and `proxy_id` settings from the Required to the
Optional section of `xpack.fleet.outputs`. CC @nchaulet since the
settings were added to the docs via
https://github.com/elastic/kibana/pull/158771). [See orange highlight]
- Adds the `ssl` setting with the `certificate` property (this setting
is already available but doesn't appear to have been documented). [See
red highlight]
- Adds the new `secrets` setting, with SSL `key` as a property. [See
blue highlight]
- Adds an example output configuration. [See green highlight]
Rel: https://github.com/elastic/ingest-docs/issues/692
See [docs
preview](https://kibana_172637.docs-preview.app.elstc.co/guide/en/kibana/master/fleet-settings-kb.html)
---
## Summary
Ref https://github.com/elastic/kibana/issues/168629
Adds a new `keep_monitoring_alive` flag to agent policies that allows
agent environments to keep the monitoring server if they choose.
> [!note]
> `monitoring_enabled: []` needs to be explicitly set for this flag to
be honored. If `monitoring_enabled` is omitted entirely, Fleet defaults
to enabling monitoring for logs and metrics.
## To test
Create a preconfigured policy where logs/metrics monitoring is
explicitly disabled and include this new flag e.g.
```yml
xpack.fleet.agentPolicies:
- name: Test preconfigured policy
id: test-preconfigured-policy
package_policies: []
monitoring_enabled: []
keep_monitoring_alive: true
```
Then, run Kibana and verify the `monitoring` block in the full agent
policy has monitoring enabled, but logs/metrics as disabled, e.g.
You can also verify the need for `monitoring_enabled: []` by creating
another preconfigured policy e.g.
```yml
- name: Test preconfigured policy 2
id: test-preconfigured-policy-2
package_policies: []
keep_monitoring_alive: true
```
This will generate a policy e.g.
All other combinations of enabling monitoring for logs/metrics should
also generate the policies as expected. If you want to verify every
combination manually:
```yml
xpack.fleet.agentPolicies:
- name: Test preconfigured policy
id: test-preconfigured-policy
package_policies: []
monitoring_enabled: []
keep_monitoring_alive: true
- name: Test preconfigured policy 2
id: test-preconfigured-policy-2
package_policies: []
keep_monitoring_alive: true
- name: Test preconfigured policy 3
id: test-preconfigured-policy-3
package_policies: []
monitoring_enabled: ['logs']
- name: Test preconfigured policy 4
id: test-preconfigured-policy-4
package_policies: []
monitoring_enabled: ['metrics']
- name: Test preconfigured policy 5
id: test-preconfigured-policy-5
package_policies: []
monitoring_enabled: ['logs', 'metrics']
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Resolves https://github.com/elastic/kibana/issues/162264
## Summary
Adds a limit on the maximum number of actions that can be queued with a
circuit breaker. The limit in serverless is set to 10,000, and 1,000,000
in the other environments.
- If a rule execution exceeds the limit, the circuit breaker kicks in
and stops triggering actions.
- Alerting rule's status updated to warning when circuit breaker is hit
Did not update the `enqueueExecution` bc it's going to be removed in
https://github.com/elastic/kibana/pull/165120.
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### To Verify
- Create a 2 rules that have actions
- Set `xpack.actions.queued.max` in kibana.yml to a low number like 2 or
3
- Use the run soon button to queue up actions and hit the circuit
breaker.
- The actions will not be scheduled and the rule status will be set to
warning
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Resolves: https://github.com/elastic/kibana/issues/162262
This PR is the backend changes to add a circuit breaker
`xpack.alerting.rules.maxScheduledPerMinute` to both serverless and
other environments that limits the number of rules to 400 runs / minute
and 10000 runs / minute, respectively. There will be another PR to
follow this one that gives the user UI hints when creating/editing rules
that go over this limit.
This circuit breaker check is applied to the following routes:
- Create Rule
- Update Rule
- Enable Rule
- Bulk Enable Rule
- Bulk Edit Rule
Also adds a new route: `/internal/alerting/rules/_schedule_frequency` to
get the current total schedules per minute (of enabled rules) and the
remaining interval allotment.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: lcawl <lcawley@elastic.co>
Co-authored-by: Xavier Mouligneau <xavier.mouligneau@elastic.co>
In this PR, I'm adding a warning message to the docs for the
`xpack.alerting.rules.run.alerts.max` setting that indicates the
consequences when setting a value higher than the default, while also
indicating it's not supported.
<img width="862" alt="Screenshot 2023-08-21 at 5 03 52 PM"
src="46a7f5d6-f6d5-475a-ab93-edf256eb9141">
cc @lcawl
---------
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
### Summary
This PR adds missing APM/Observability settings to the documentation:
- `observability:apmAgentExplorerView`
- `observability:apmAWSLambdaPriceFactor`
- `observability:apmAWSLambdaRequestCostPerMillion`
- `observability:apmEnableContinuousRollups`
- `observability:apmEnableServiceMetrics`
- `observability:apmLabsButton`
- `observability:apmServiceGroupMaxNumberOfServices`
- `observability:apmDefaultServiceEnvironment`
This PR also adds @elastic/obs-docs as a codeowner to
`/x-pack/plugins/observability/server/ui_settings.ts` so that we don't
miss documenting settings moving forward.
Closes https://github.com/elastic/kibana/issues/118795.
