## Summary
While investigating some issues with the alertsClient, I realized that
we weren't writing out any rule information for the logged messages.
This made debugging quite difficult, as I wanted to see the rule, so had
to search through the alerts indices for the specified alert to get it's
rule id, rule type, etc.
As an example, see https://github.com/elastic/kibana/issues/190376
This PR adds that kind of rule info to the logged messages in
alertsClient, as well as the typical sort of tags we write out (rule id,
rule type, module).
## Summary
This PR is a precursor to https://github.com/elastic/kibana/pull/189871,
as part of the spaces improvement initiative there's a need to be able
to share the user privilege assignment component between the roles
experience and the new spaces experience to prevent duplication of
business logic and cohesiveness in the privilege assignment experience.
The aforementioned PR extracts the required component into it's own
package so it might be consumed as needed, this PR is particularly
concerned with extracting business logic said UI component depends on
that exists still within the security plugin. For context; the security
plugin already depends on the spaces plugin, so having the spaces plugin
in turn statically depend on the security plugin creates a cyclic
dependency. That being said to complement the eventual state of said
component so it might be imported elsewhere outside of the security
plugin there's a need to extract further logic into standalone packages,
so that the spaces plugin can consume this plugin without the afore
mentioned cyclic dependency problem.
#### Visually;
##### Problem;
##### Proposal
[^legend]
[^legend]: items marked in blue are the packages created in this PR,
whilst the entire diagram is the proposed future state
<!--### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
-->
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Change the ownership from `@elastic/kibana-core` to
`@elastic/appex-ai-infra`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fixes `source` node parsing. Correctly handles cluster part and
unescapes the quoted index string part.
1. First removes the cluster string part.
2. Unquotes and unescapes the index string part (if it is quoted and
escaped).
Those two were not done before: the index patter string was unquoted as
a whole (with cluster part attached); and, the index string was not
unescaped.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### For maintainers
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Closes#190995
## Summary
This PR adds grouping functionality to the alerts page alert table based
on @umbopepato's implementation in this [draft
PR](https://github.com/elastic/kibana/pull/183114) (basically, he
implemented the feature and I adjusted a bit for our use case :D).
For now, we only added the **rule** and **source** as default grouping,
and I will create a ticket to add tags as well. The challenge with tags
is that since it is an array, the value of the alert is joined by a
comma as the group, which does not match with what we want for tags.
Here is how we show the rules that don't have a group by field selected
for them: (We used "ungrouped" similar to what we have in SLOs)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: DeDe Morton <dede.morton@elastic.co>
Co-authored-by: Shahzad <shahzad31comp@gmail.com>
Also txt, rtf, doc, docx, xls, xlsx, ppt, pptx, odt, ods, and odp.
Adds the ability to automatically add a semantic text field to the
mappings and a `copy_to` processor to duplicate the field. This is
needed for the mappings generated for the attachment processor which
adds a nested `attachment.content` field which cannot be used as a
semantic text field.
After a successful import, a link to Search's Playground app is shown.
Navigating there lets the user instantly query the newly uploaded file.
https://github.com/user-attachments/assets/09b20a5f-0e02-47fa-885e-0ed21374cc60
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
## Summary
Part of the OTel effort. This PR adds otel datastream patterns into the
default indices that are used by the APM UI.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Caue Marcondes <caue.marcondes@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Closes https://github.com/elastic/kibana/issues/189802.
This PR aims to use a dockerized package registry version for testing.
In order to test it locally you have to set the value of
`FLEET_PACKAGE_REGISTRY_PORT` env var in your terminal, and you also
need to have a docker daemon running.
For example, you can open a terminal and start the server
```
node x-pack/plugins/observability_solution/dataset_quality/scripts/api --server
```
then open a new terminal set the var value and start the runner with the
specific test using this configuration
```
export set FLEET_PACKAGE_REGISTRY_PORT=12345
node x-pack/plugins/observability_solution/dataset_quality/scripts/api --runner --grep-files=integrations
```
If you want to test again without the dockerized version, you should
remove the value of the var
```
unset FLEET_PACKAGE_REGISTRY_PORT
```
## Summary
When Kibana tries to authenticate a request that already has an
`Authorization` header (not a cookie, client certificate, or Kerberos
ticket), the authentication is performed by the [HTTP authentication
provider](https://www.elastic.co/guide/en/kibana/current/kibana-authentication.html#http-authentication).
Unlike session/Kerberos/PKI providers, this provider returns an
authentication result that doesn't explicitly tell Core which
authorization headers should be used (e.g., `t.authenticated({ state:
authenticationResult.user, --> requestHeaders:
authenticationResult.authHeaders <-- ... });`), assuming that Core will
just use the headers from the request. The `Authorization` header is
forwarded to Elasticsearch by default, no additional configuration is
required.
This worked well previously, but I think with the introduction of the
the
[`getSecondaryAuthHeaders`](https://github.com/elastic/kibana/pull/184901)
method this is the first time where this assumption doesn't hold.
Internally, this method tries to reuse authentication headers that were
provided to Core by the authentication provider during the request
authentication stage — headers that the HTTP authentication provider
never provided before.
This PR makes the HTTP authentication provider behave consistently with
the rest of the providers we support today.
## Summary
This adds a `samplesFormat` group to the API. This group is filled out
by the frontend when parsing the provided samples and used to set the
log parsing specification for the produced integration.
We check this parameter to add toggle to support multiline
newline-delimited JSON in the filestream input.
## Release note
Automatic Import now supports the 'multiline newline-delimited JSON' log
sample format for the Filestream input.
## Detailed Explanation
We add the optional `samplesFormat` group to the API, consisting of
- `name`,
- (optional) `multiline`,
- and (optional) `json_path`.
Example values of this parameter:
- `{ name: 'ndjson', multiline: false }` for a newline-delimited JSON,
known as [NDJSON](https://github.com/ndjson/ndjson-spec) (where each
entry only takes one line)
- `{ name: 'ndjson', multiline: true }` for newline-delimited JSON where
each entry can span multiline lines
- `{ name: 'json', json_path: [] }` for valid JSON with the structure
`[{"key": "message1"}, {"key": "message2"}]`
- `{ name: 'json', json_path: ['events'] }` for valid JSON with the
structure `{"events": [{"key": "message1"}, {"key": "message2"}]}`
The `json_path` parameter is only relevant for `name: 'json'` and refers
to the path in the original JSON to the array representing the events to
ingest. Currently only one level is recognized:
Not all combinations of a log format with input type will work; more
supported combinations as well as better user feedback on unsupported
combinations will come later (see
https://github.com/elastic/security-team/issues/10290).
In this PR we add support for the multiline NDJSON format for the
`fileinput` input type. This support comes in the form of the
user-changeable toggle under "Advanced Settings" that will be set to on
in cases where we multiline NDJSON format
---------
Co-authored-by: Marius Iversen <marius.iversen@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
### test instructions
1. run kibana with `yarn start --run-examples`
2. navigate to the "Controls" developer example
3. Click "Add new data-control" button. Notice how the menu sets "width"
to medium and selects "grow".
4. Add new control, setting "width" to small and unselected "grow".
Click "Save and close".
5. Click "Add new data-control" button. Notice how the menu sets "width"
and "grow" to last used values.
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Hannah Mudge <Heenawter@users.noreply.github.com>
## Summary
Use typescript's async processes to start quick checks in parallel 👍
Check out these for runs:
- happy case:
https://buildkite.com/elastic/kibana-pull-request/builds/227443#01914ca3-1f0d-4178-b539-263fbc588e98
- some broken checks:
https://buildkite.com/elastic/kibana-pull-request/builds/228957#01917607-f7bd-4e08-8c70-7fdc3f9c12d1
Benefits:
- with this (+more CPU) we can speed up the quick-check step's runtime,
from ~15m to ~7m.
- the added benefit is that all checks run so that we won't bail on the
1st one
Disadvantage:
- uglier error output, since we collect the logs asynchronously, and
print it only upon failure
- ~no output printed for happy checks (can be changed)~
Extra:
- additionally, `yarn quick-checks` will now allow devs to run these
checks locally (adjustments made so that the checks won't fail in local
dev)
- added the option to declare a 'context' for tooling loggers, so we can
identify which script logs
Solves 2/3 of https://github.com/elastic/kibana-operations/issues/124
(+speedup)
## Summary
A missed catch+log during the fast transformation of typescript files to
js files caused the task pooling library, `piscina` to terminate without
a warning. The terminated workers would crash with unexpected messages,
and that resulted in original errors to be lost, and misleading error
messages. This PR adds some logging.
See:
https://buildkite.com/elastic/kibana-pull-request/builds/228663#019170d4-4f00-4ccc-915e-7d8b0a6ca020
- the original causes were incorrect `await` usages, but that was never
revealed, because we only got the messages that arrived to a terminated
worker.
See also, for more background:
https://elastic.slack.com/archives/C5UDAFZQU/p1724236149083029
## Summary
Allows to set context for the tooling logger. The context would be
logged before the log output, like so:
```
info [My-Tool] Hello!
```
Split from: https://github.com/elastic/kibana/pull/190401
---------
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
This PR contains the following updates:
| Package | Type | Update | Change |
|---|---|---|---|
| [chromedriver](https://togithub.com/giggio/node-chromedriver) |
devDependencies | patch | [`^127.0.2` ->
`^127.0.3`](https://renovatebot.com/diffs/npm/chromedriver/127.0.3/127.0.3)
|
---
### Configuration
📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR has been generated by [Renovate
Bot](https://togithub.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40MjUuMSIsInVwZGF0ZWRJblZlciI6IjM3LjQyNS4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJUZWFtOk9wZXJhdGlvbnMiLCJyZWxlYXNlX25vdGU6c2tpcCJdfQ==-->
Co-authored-by: elastic-renovate-prod[bot] <174716857+elastic-renovate-prod[bot]@users.noreply.github.com>
## Summary
close https://github.com/elastic/kibana/issues/183687
## Feature
- Implement dashboard view stats UI on top of usage counter that counts
dashboard views for last 90 day and shows weekly histogram.
- (Even if there is not a lot of data, we still show it as a weekly
histogram, so it can be pretty empty intially)
## Implementation
### Server side
Dashboard plugin registers new routes to increase the view count and get
stats. Routes are protected for users with dashboard access only. The
implementation is located in
`@kbn/content-management-content-insights-server` and internally uses
usage counters. The retention is 90 days, so we can only show stats for
last 90 days.
### Client side
- Dashboard uses the client from
`@kbn/content-management-content-insights-public` to increase the view
count every time a user opens a dashboard.
- TableListView opens the flyout from
`@kbn/content-management-content-insights-public`to display the stats
## How to test
- For new views just open a dashboard and check that view stat is
increased
- For old views you can populate the usage counters with historic data.
I used the following script:
https://gist.github.com/Dosant/425042fcf75d5e40e5a46374f6234a54
Resolves https://github.com/elastic/kibana/issues/190994
### Test instructions
1. create new canvas workpad
2. Click "Select type" and select "Maps"
3. Add a documents layer to your map and click "Save and return"
4. Click "canvas" bread crumb to go back to canvas listing page
5. Open you work pad again. Map should still display data
## Summary
This PR offers small updates to documentation on:
* Calling Kibana APIs to automate report generation
* Using Elasticsearch APIs directly to export data
In this PR, I'm renaming the task managers as we prepare to rollout the
`mget` task claiming strategy as the default.
Rename:
- `unsafe_mget` -> `mget`
- `default` -> `update_by_query`
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Addresses https://github.com/elastic/kibana/issues/179549
Relates to https://github.com/elastic/kibana/pull/183113
## Update
Since the Appex QA team has taken on deployment agnostic tests, a lot of
the original implementation of this PR has changed. Now that the Appex
QA team has provided a current directly to write deployment agnostic
tests, the burn rate rule tests have been moved here.
To finish onboarding the burn rate rule test to this new framework, the
following was done.
1. Add an `oblt.stateful.config.ts` file to complement the existing
`oblt.serverless.config.ts` file to ensure the tests are run in CI
2. Ensure our test config is added to the buildkite pipepline
3. Add the alerting service to the new `deployment_agnostic/services`
directory.
4. Port the tests over to the new `deployment_agnostic` directory
To run serverless
```
node scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts
node scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts --grep="Burn rate rule"
```
To run stateful
```
node scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts
node scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts --grep="Burn rate rule"
```
For context, I've kept the history from the original PR description
below.
## 🍒 History
A new type of config file will be allowed for API integration and
functional tests within the `x-pack/test` folder, using a pattern of
`*.serverless.config.ts` — these config files will specify configuration
needed to run a set of tests in a serverless deployment context.
FTR tests already make use of the `it.tags(['my-tag-1', 'my-tag-2'])`
pattern, and so we would like to stay with that pattern rather than
introducing a new type of mocha tag in the it block's "description" as
it was introduced in this
[POC](https://github.com/elastic/kibana/pull/183113). The difference
with the previous PR in terms of tagging is that we use `suiteTags`
instead of `mochaOps`
Adding following in the config files:
**serverless config**
```
suiteTags: {
include: ['serverless'],
},
```
**ess config**
```
suiteTags: {
include: ['ess'],
},
```
and then adding `this.tags(['serverless', 'ess'])` in the test suite
instructs the test runner to run the same test suite in both
environments.
In order to keep things simple, we stay with the current skip approach,
which means that flaky tests will be skipped for all environments by
appending .skip() to the suite or to specific test cases.
## Description
- This PR uses `suiteTags` for tagging the tests appropriately. We
decide through following labels in which environment the tests are going
to be executed:
- **@ess**: Runs in an ESS environment (on-prem installation) as part of
the CI validation on PRs.
- **@serverless**: Runs in a serverless environment.
- It introduces a new folder
`x-pack/test/observability_solution_api_integration` which will serve as
a centralized location for all tests by obs-ux-management team that must
be run in Serverless and ESS environments. A list of all tests can be
found in the R&D
[issue](https://github.com/elastic/kibana/issues/179549)
- Within this folder, there is a "**config**" subdirectory that stores
base configurations specific to both the Serverless and ESS
environments. These configurations build upon the base configuration
provided by test_serverless and api_integration, incorporating
additional settings such as environment variables and tagging options.
- The file
`x-pack/test/observability_solution_api_integration/test_suites/alerting/burn_rate/burn_rate_rule.ts`
is functional in both Serverless and ESS
- It removes the existing burn rate rule from
`x-pack/test_serverless/api_integration/test_suites/observability/alerting/burn_rate/burn_rate_rule.ts`
- The `alertingApi` and `sloApi` services are moved to
`test/api_integration` servers
In the screenshot below you can see the `test_suites` folder structure,
after having migrated the current slo burn rate rule. We recommend
having an `alerting` and `slo` subfolders. Rest observability apps could
be added as another subfolder under test_suites. As part of this PR, the
`alerting > burn_rate` subfolders are created.
<img width="376" alt="Screenshot 2024-05-13 at 09 21 28"
src="3ccaf0a5-1443-4bad-ad06-daa347488bf1">
## How to run locally
You can navigate into the new `observability_solution_api_integration`
folder and use following commands to run the tests in serverless and ess
environments accordingly. You can find more information in the README
file of the observability_solution_api_integration folder.
```
cd x-pack/test/observability_solution_api_integration
// SERVERLESS
npm run alerting_burn_rate:server:serverless
npm run alerting_burn_rate🏃serverless
// ESS
npm run alerting_burn_rate:server:ess
npm run alerting_burn_rate🏃ess
```
## CI
- It includes a new entry in the `ftr_configs.yml` to execute the newly
added tests in the pipeline.
- It involves the addition of `suiteTags` in both
serverless/config.base.ts and ess/config.base.ts. In the case of
serverless, it includes **@serverless** while excluding
**@skipInServerless**. Similarly, for ess, it includes **@ess** and
excludes **@skipInEss**.
## Quality Gates and MKI pipeline
The Platform team will support config files within `x-pack/test` folder
with a pattern of `*.serverless.config.ts`, so these tests will be
included in Kibana's Quality gates and will be run against a real MKI
environment.
---------
Co-authored-by: Dominique Belcher <dominique.clarke@elastic.co>
Co-authored-by: Dominique Clarke <doclarke71@gmail.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
Co-authored-by: Robert Oskamp <traeluki@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Ref https://github.com/elastic/kibana/pull/180922
Re-add the `agent.monitoring.enabled` and
`agent.monitoring.buffer.enabled` settings under Agent Policy ->
Advanced Settings.
cc @pierrehilbert
## Summary
We want to allow rule executors to report warnings back to the framework
and have them show up in the stack management UI. We currently have a
mechanism for this that is used by detection rules but they handle how
to show the warning in their custom UI so the warning does not appear in
the stack management UI. This PR adds a new value to the allow list of
"warning reasons". This should not warrant a schema version bump because
it is a new value for an existing enum but we need to do an intermediate
release so the new schema is recognized before we start setting it
anywhere. This PR is just for schema changes. PR that uses these schema
changes can be found here: https://github.com/elastic/kibana/pull/189312
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
