Adds a new `tags` property to our ML Modules which contains an array of
strings to used to filter the modules returned when calling
`/internal/ml/modules/get_module` or `/internal/ml/modules/recognize`
Adds a new kibana config setting `xpack.ml.compatibleModuleType` which
will enforce a module filter.
This setting supports the values `security`, `observability` or `search`
and will be used by the serverless projects to ensure only modules
relevant to the current project are discoverable.
When matching against the `xpack.ml.compatibleModuleType` setting,
modules with no tags will be returned along with modules which match the
`compatibleModuleType` value.
The endpoints `/internal/ml/modules/get_module` and
`/internal/ml/modules/recognize` can also take a `filter` query
parameter in the form of a comma separated list of values. If any of
these filter values match a module's tags, the module will be returned.
Modules with no tags will not be matched when using the `filter`
parameter.
This PR also updates the security plugin to apply a `security` filter
when retrieving modules and running recognize. This will improve
performance, especially for recognize which runs the queries from every
module on the supplied index pattern.
Examples
**Running the Security serverless project.**
Request:
`/internal/ml/modules/get_module`
Response:
All modules containing `tags: ["security"]`
Plus all modules with no `tags` or `tags: []`
Request:
`/internal/ml/modules/get_module?filter=observability`
Response:
An empty list
**Running stateful es**
Request:
`/internal/ml/modules/get_module`
Response:
All modules
Request:
`/internal/ml/modules/get_module?filter=security`
Response:
Only modules containing `tags: ["security"]`
Request:
`/internal/ml/modules/get_module?filter=security,observability`
Response:
Modules containing `tags: ["security"]` and `tags: ["observability"]`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Branch with Infra UI enabled in Serverless.
This exists for us to have a working environment based on main, but with
infra turned on. We could merge this at a later date when we plan to
enable Infra on serverless.
Fixes https://github.com/elastic/kibana/issues/167850Fixes#168065Fixes#168853
Fixes https://github.com/elastic/kibana/issues/167654
## Summary
This PR removes the Storage size column in Data stream in serverless as
the data stream stats API on serverless doesn't currently return the
storage size.
### How to test:
Verify that the column is not displayed in serverless:
1. Start Es with `yarn es serverless` and Kibana with `yarn
serverless-{es/oblt/security}`
2. Go to Stack Management -> Index Management -> Data Streams
3. Switch on the "Include stats" toggle
4. Verify that the "Storage size" column is not shown in the table.
Verify that the column is displayed in stateful:
1. Start Es with `yarn es snapshot` and Kibana with `yarn start`
2. Go to Stack Management -> Index Management -> Data Streams
3. Switch on the "Include stats" toggle
4. Verify that the "Storage size" column is shown in the table.
<!---
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
-->
## Summary
Fixes https://github.com/elastic/kibana/issues/165895
This PR limits which index settings are displayed on the index details
page, "Settings" tab in the edit mode. On serverless only a handful of
index settings will be editable by the user. The UI only prevents
displaying some index settings, but it's still possible for the user to
type in a setting that can't be edited. That is the case on dedicated as
well.
### How to test
1. Start Serverless ES and Kibana
2. Navigate to Index Management and create a test index
3. Click on the index name and on the details page click the tab
"Settings"
4. Toggle the "Edit mode" switch and verify that only editable settings
are displayed.
#### Screenshot
<img width="527" alt="Screenshot 2023-10-16 at 20 25 49"
src="e6678cca-3494-4c63-ae66-ace9c823d12d">
## Summary
Closes https://github.com/elastic/ingest-dev/issues/2396
Added a new kibana task that publishes Agent metrics every minute to
data streams installed by fleet_server package.
Opened the pr for review, there are a few things to finalize, but the
core logic won't change much.
To test locally:
- Install fleet_server package 1.4.0 from
[this](https://github.com/elastic/integrations/pull/8145) pr to get the
mappings
- Start kibana locally, wait for a few minutes for the metrics task to
run (every minute)
- Go to discover, `metrics-*` index pattern, filter on
`data_stream.dataset: fleet_server.*`
- Expect data to be populated in `fleet_server.agent_status` and
`fleet_server.agent_versions` datasets.
<img width="1787" alt="image"
src="615af9df-fe4b-4c17-8c8c-88646c403a18">
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
fix https://github.com/elastic/kibana/issues/166233 (this is a short
term fix that removes "analytics" references from the empty state page
in serverless observability analytics apps)
## Summary
This PR implements a standard way to have different static settings for
the serverless and ess (stateful) environments. It centralizes flags,
which were set using different approaches previously, in a single
configuration.
This aims to make it easier for developers to enable/disable parts of
the application in serverless projects.
Default:
```
sideNavEnabled: true,
ILMEnabled: true,
ESQLEnabled: true,
```
Serverless:
```
xpack.securitySolution.offeringSettings: {
sideNavEnabled: false, # Internal security side navigation disabled, the serverless global chrome navigation is used instead
ILMEnabled: false, # Index Lifecycle Management (ILM) functionalities disabled, not supported by serverless Elasticsearch
ESQLEnabled: false, # ES|QL disabled, not supported by serverless Elasticsearch
}
```
### Consume the settings
#### Server
- Plugin parsed `ConfigType`:
`this.config.settings.ESQLEnabled`
#### UI
- Plugin attribute:
`this.configSettings.ESQLEnabled`.
- Components can access it from Kibana services:
`useKibana().services.configSettings.ESQLEnabled;`
---------
Co-authored-by: Vitalii Dmyterko <92328789+vitaliidm@users.noreply.github.com>
## Summary
This PR leverages the work done
[here](https://github.com/elastic/kibana/pull/165596) to introduce ES|QL
into timeline. The goal of this PR is to provide security users easy
access to ESQL from within the security solution. It will be released in
`technical preview` for the 8.11 release.
<img width="1725" alt="image"
src="0e275cf7-bbce-476b-b1dc-8936427ad14f">
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Jatin Kathuria <jatin.kathuria@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Jatin Kathuria <jtn.kathuria@gmail.com>
Switches to using the serverless config file to enabled/disable ML
features rather than a function shared from the setup contract.
Storing these flags in a config file means they are already available
when setup runs and so can be used when registering integrations into
other plugins.
Removes the dependency on ML from `security_solution_serverless`,
`serverless_observability` and `serverless_search`
## Summary
Updated the grok debugger and painless lab plugins to handle the
`enabled` config flag so they can easily be disabled for serverless
search.
### Screenshots
Before:
After:
<img width="1914" alt="image"
src="ad48ba1b-0f1d-4da3-aeac-4868781f6832">
Resolves https://github.com/elastic/kibana/issues/162264
## Summary
Adds a limit on the maximum number of actions that can be queued with a
circuit breaker. The limit in serverless is set to 10,000, and 1,000,000
in the other environments.
- If a rule execution exceeds the limit, the circuit breaker kicks in
and stops triggering actions.
- Alerting rule's status updated to warning when circuit breaker is hit
Did not update the `enqueueExecution` bc it's going to be removed in
https://github.com/elastic/kibana/pull/165120.
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### To Verify
- Create a 2 rules that have actions
- Set `xpack.actions.queued.max` in kibana.yml to a low number like 2 or
3
- Use the run soon button to queue up actions and hit the circuit
breaker.
- The actions will not be scheduled and the rule status will be set to
warning
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Resolves: https://github.com/elastic/kibana/issues/162262
This PR is the backend changes to add a circuit breaker
`xpack.alerting.rules.maxScheduledPerMinute` to both serverless and
other environments that limits the number of rules to 400 runs / minute
and 10000 runs / minute, respectively. There will be another PR to
follow this one that gives the user UI hints when creating/editing rules
that go over this limit.
This circuit breaker check is applied to the following routes:
- Create Rule
- Update Rule
- Enable Rule
- Bulk Enable Rule
- Bulk Edit Rule
Also adds a new route: `/internal/alerting/rules/_schedule_frequency` to
get the current total schedules per minute (of enabled rules) and the
remaining interval allotment.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: lcawl <lcawley@elastic.co>
Co-authored-by: Xavier Mouligneau <xavier.mouligneau@elastic.co>
## Summary
This moves connectors functions to a shared package so Serverless Search
and Enterprise Search can both use it.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/137810
Part of https://github.com/elastic/kibana/issues/163248
This PR is the **first iteration** of the new ES|QL language in kibana.
The majority of the functionality is based on the existing functionality
for SQL (which has already been merged). This builds on top of it to
enable the functionality for ES|QL. We decided to remove SQL for now so
this PR enables ES|QL and hides SQL. We are not removing the code for
now.
### Important notes:
- This PR contains basic autocomplete functionality for the ES|QL
commands. We want to improve it in follow up PRs
- The majority of the tests for SQL were moved to work with ES|QL
instead
- The search strategy is a very simple endpoint for now as we don't have
async search nor pagination
- Now that we remove SQL, the ui for ES|QL selection has changed
- The documentation for ESQL has been handled by the docs team so it is
already reviewed and in sync with the official documentation
- ES|QL is disabled in serverless projects for now
### Changes from SQL:
- The Discover histogram now is being created with ES|QL (using the
date_trunc function). This gives it the ability to be saved on a
dashboard and also to be edited inline.
- ES|QL sometimes returns some warnings (on the search headers). For
example when we are trying to date parse a string that doesnt contain a
valid date). These warnings are also reported on the UI. There is a bug
in ES and the warning doesnt come always
https://github.com/elastic/elasticsearch-internal/issues/1465)
- We have 2 types of commands:
- Transformational commands (keep, stats) -> when they exist in the
query Discover should render the Table view (selected columns)
- Non transformational commands (all the others) -> in that case
Discover renders the Document view
- ESQL switch on advanced settings is now on by default
**Discover view with non transformational commands**
<img width="1678" alt="image"
src="abe100e1-01e9-4fe0-9b89-6d8bdf6443fc">
**Discover view with transformational commands**
<img width="1679" alt="image"
src="e46af422-daeb-4be5-88cf-522211674ff5">
### Missing
- ESQL autocomplete is not perfect, we are going to work on it on a
future PR. Specifically:
- There is a bug in autocomplete, sometimes writing a query overwrites
the existing one, will deal with it in a future PR
- Further improvements
### Checklist
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Alexey Antonov <alexwizp@gmail.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>
Co-authored-by: Peter Pisljar <peter.pisljar@elastic.co>
Co-authored-by: Marco Liberati <dej611@users.noreply.github.com>
Co-authored-by: Matthias Wilhelm <matthias.wilhelm@elastic.co>
Co-authored-by: Davis McPhee <davis.mcphee@elastic.co>
## Summary
Enable ZDT migrations to run on `migrator` node roles (the default).
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Issues when rolling out Kibana's | Med. | Low | It's possible this
change could lead to interruption in internal Elastic environments |
This adds an empty connectors page to the Elasticsearch project. The
Enterprise Search plugin exports relevant connectors data and functions,
the Search plugin simply consumes these.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Closes#161337
## Summary
Uses build flavor(see #161930) to disable specific Kibana security,
spaces, and encrypted saved objects HTTP API routes in serverless (see
details in #161337). HTTP APIs that will be public in serverless have
been handled in #162523.
**IMPORTANT: This PR leaves login, user, and role routes enabled. The
primary reason for this is due to several testing mechanisms that rely
on basic authentication and custom roles (UI, Cypress). These tests will
be modified to use SAML authentication and serverless roles in the
immediate future. Once this occurs, we will disable these routes.**
### Testing
This PR also implements testing API access in serverless.
- The testing strategy for disabled routes in serverless is to verify a
`404 not found `response.
- The testing strategy for internal access routes in serverless is to
verify that without the internal request header
(`x-elastic-internal-origin`), a `400 bad request response` is received,
then verify that with the internal request header, a `200 ok response`
is received.
- The strategy for public routes in serverless is to verify a `200 ok`
or `203 redirect` is received.
~~blocked by #161930~~
~~blocked by #162149 for test implementation~~
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@elastic.co>
Part of https://github.com/elastic/kibana/issues/155764.
In this PR, I'm setting the
`xpack.task_manager.allow_reading_invalid_state` serverless setting to
`false` so Kibana doesn't allow reading invalid state when loading
tasks.
I'm also doing the same for the functional tests to ensure valid task
state is always read.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/163340
This PR disables graph in serverless. Specifically:
- creates a serverless yml setting for disabling graph
- adds the setting in serverless.yml
## Summary
fixes https://github.com/elastic/kibana/issues/160337
This PR
- hides cases in the serverless Elasticsearch project, cases APIs throw
error
- throws 403 from API when `owner=cases` for security or observability
serverless mode
- verifies the behaviour in serverless functional as well as
api_integration tests
**How to test**
- Boot up `es` serverless solution and make sure that `cases` from the
navbar is hidden and cannot not be accessible through url as well
- Boot up `observability` or `security` serverless solutions and make
sure that `cases` is available in the navbar and works fine
- Boot up classic kibana and make sure that the left navbar has the same
menu entries it always had.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
20c1974e-44f0-45b0-80aa-e644fec148ff
### For maintainers
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Reverts elastic/kibana#159674
The Elasticsearch team has changed their guidance about `fast_refresh`
and want this setting to be applied from within an Elasticsearch plugin
## Summary
Adds the necessary yml settings to the serverless config. These settings
are making ui changes in dashboards / visualizations editors and they
apply for all projects.
For testing run `yarn start --serverless=es` and check that the ui is
working as expected with the legacy editors (TSVB, agg based) in a read
only mode.
Starting in Node.js 17 the IP address from a DNS lookup resolves in the
order returned from the resolver. Prior versions resolve with IPv4
addresses sorted before IPv6 addresses.
This restores the previous behavior. In distributions, this can be
configured via the `--dns-result-order` flag in `node.options`.
Closes#163006
[Node.js 17
changelog](https://github.com/nodejs/node/blob/main/doc/changelogs/CHANGELOG_V17.md)
[Documentation](https://nodejs.org/docs/latest-v18.x/api/cli.html#--dns-result-orderorder)
Testing
1) Resolve localhost to 127.0.0.1 and then ::1 in `/etc/hosts`
2) `yarn start`. Logs should indicate connecting to elasticsearch over
127.0.0.1:9200
## Summary
Closes https://github.com/elastic/kibana/issues/160141
The avatar menu needs to be displayed for serverless. It was previously
required to be hidden in serverless, so a config 'showNavLinks' was
added. This config is no longer needed, so it has been removed.
## Testing
Start KB with the `--serverless` flag and login as `elastic`.
The Avatar should appear in the top right coner.
## Summary
This PR sets up the reporting plugin for the serverless implementation
by adding properties to the existing reporting config.
Image reporting is enabled for dev mode but disabled for serverless.
Canvas is disabled for serverless.
## To Test
Run `yarn es snapshot --license trial` in one terminal and then `yarn
start`. Load sample data and you should be able to see the option to
have PDF and PNG reports in Dashboard's Share Menu.
Now run `yarn es snapshot --license trial` and `yarn serverless-es`. You
should see that Dashboard's share menu does not include PDF or PNG
Reports. However there is still the option to see run CSV reports and
see the Reporting in Management.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Timothy Sullivan <tsullivan@elastic.co>
