## Summary
Follow up of [first CDN
PR](https://github.com/elastic/kibana/pull/169408). Primary focus is
replacing our build nr with SHA that allows cache busting and maintains
anti-collision properties.
## How to test
Start Kibana as usual navigating around the app with the network tab
open in your browser of choice. Keep an eye out for any asset loading
errors. It's tricky to test every possible asset since there are many
permutations, but generally navigating around Kibana should work exactly
as it did before regarding loading bundles and assets.
## Notes
* did a high-level audit of usages of `buildNum` in `packages`, `src`
and `x-pack` adding comments where appropriate.
* In non-distributable builds (like dev) static asset paths will be
prefixed with `XXXXXXXXXXXX` instead of Node's `Number.MAX_SAFE_INTEGER`
* Added some validation to ensure the CDN url is of the expected form:
nothing trailing the pathname
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### Risk Matrix
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| We break some first or third party dependencies on existing asset
routes | Med | High | Attempting to mitgate by serving static assets
from both old and new paths where paths have updated to include the
build SHA. Additioanlly: it is very bad practice to rely on the values
of the static paths, but someone might be |
| Cache-busting is more aggressive | High | Low | Unlikely to be a big
problem, but we are not scoping more static assets to a SHA and so every
new Kibana release will require clients to, for example, download fonts
again. |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Once we're moving to the elastic-wide buildkite agents, and away from
the kibana-buildkite-managed ones, we won't have default access to the
buckets we used to use, as the assumed service account will differ.
**Note:** Although this will only be required in the new infra, but this
change can be merged and expected to work properly in the current infra
as well.
### Solution
We've set up a central service-account with rights to impersonate other
service accounts that have controlled access to individual buckets to
minimize the reach and influence of individual accounts. See:
https://github.com/elastic/kibana-operations/pull/51
**several of the changes weren't tested, as they're part of CI tasks
outside the PR build** - will merge with caution and monitor the
stability afterwards
TODO: _add access, and assume account before other GCS bucket usages_
- [x] storybook
- [x] coverage
(.buildkite/scripts/steps/code_coverage/reporting/uploadPrevSha.sh)
- [x] upload static site
(.buildkite/scripts/steps/code_coverage/reporting/uploadStaticSite.sh)
- [x] SO object migration
(.buildkite/scripts/steps/archive_so_migration_snapshot.sh)
- [x] ES Snapshot manifest upload
(.buildkite/scripts/steps/es_snapshots/create_manifest.ts)
- [x] Scalability?
(.buildkite/scripts/steps/functional/scalability_dataset_extraction.sh)
- [x] Benchmarking
(.buildkite/scripts/steps/scalability/benchmarking.sh)
- [x] Webpack bundle analyzer
(.buildkite/scripts/steps/webpack_bundle_analyzer/upload.ts)
- [x] ~Build chromium (x-pack/build_chromium/build.py)~ Not needed, as
it's manual, and not a CI task
TODO: _others_
- [x] Remove manifest upload
(.buildkite/scripts/steps/es_serverless/promote_es_serverless_image.sh)
- [x] Decide if we should merge with the CDN access: no, SRE is managing
that account
- [x] Bazel remote cache seems to also rely on gcs - roles PR:
https://github.com/elastic/kibana-operations/pull/56
Closes: https://github.com/elastic/kibana-operations/issues/29
Part of: https://github.com/elastic/kibana-operations/issues/15
## Summary
It closes#173405
This PR adds support to load Grouping components from URL parameters by
the following changes:
### Grouping package changes:
- Updated the `onChange` handler in the `useGrouping` hook to retrieve
all the selected groupByFields.
- Updated the `onChange` handler in the `useGrouping` hook to trigger
when a group is selected or unselected.
- Added check to not send Telemetry when a group is unselected (this
keeps current behavior while triggering onChange for unselected groups.)
### Cloud Security changes:
- Updated `use_cloud_security_grouping` hook to check for the `groupBy`
URL params on the initial load to update the grouping component.
- Updated `use_cloud_security_grouping` hook to update the URL params
whenever the groups changes.
## Recording
965dfc5a-0e4e-4de8-baba-aa502ab5a010
## Summary
This PR changes the SLO creation behavior via the UI. Instead of having
a checkbox to create the Burn Rate rule, with this PR, the Burn Rate
Rule will be created by default. The Burn Rate rule is only created by
default when using the UI, the create SLO API does not create a rule by
default.
## Summary
### Fleet Changes:
- Two new utilities that return `AsyncIterator`'s:
- one for working with ElasticSearch `.search()` method
- one for working with SavedObjects `.find()` method
- NOTE: although the `SavedObjects` client already supports getting back
an `find` interface that returns an `AysncIterable`, I was not
convenient to use in our use cases where we are returning the data from
the SO back to an external consumer (services exposed by Fleet). We need
to be able to first process the data out of the SO before returning it
to the consumer, thus having this utility facilitates that.
- both handle looping through ALL data in a given query (even if >10k)
- new `fetchAllArtifacts()` method in `ArtifactsClient`: Returns an
`AsyncIterator` enabling one to loop through all artifacts (even if
>10k)
- new `fetchAllItemIds()` method in `PackagePolicyService`: return an
`AsyncIterator` enabling one to loop through all item IDs (even if >10k)
- new `fetchAllItems()` method in `PackagePolicyService`: returns an
`AsyncIterator` enabling one to loop through all package policies (even
if >10k)
### Endpoint Changes:
- Retrieval of existing artifacts as well as list of all policies and
policy IDs now use new methods introduced into fleet services (above)
- Added new config property -
`xpack.securitySolution.packagerTaskTimeout` - to enable customer to
adjust the timeout value for how long the artifact packager task can
run. Default has been set to `20m`
- Efficiencies around batch processing of updates to Policies and
artifact creation
- improved logging
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
Resolves https://github.com/elastic/kibana/issues/173009
## Summary
This PR:
- Changes the field name from `notification_delay` to `alert_delay`
- Updates the alerts client and rule registry to index new alert docs on
a delay
- Updates the framework code to delay the creation of an alert
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### To verify
- Use [Dev Tools](http://localhost:5601/app/dev_tools#/console) to
create a rule with the `alertDelay`
```
POST kbn:/api/alerting/rule
{
"params": {
"searchType": "esQuery",
"timeWindowSize": 5,
"timeWindowUnit": "m",
"threshold": [
-1
],
"thresholdComparator": ">",
"size": 100,
"esQuery": """{
"query":{
"match_all" : {}
}
}""",
"aggType": "count",
"groupBy": "all",
"termSize": 5,
"excludeHitsFromPreviousRun": false,
"sourceFields": [],
"index": [
".kibana-event-log*"
],
"timeField": "@timestamp"
},
"consumer": "stackAlerts",
"schedule": {
"interval": "1m"
},
"tags": [],
"name": "test",
"rule_type_id": ".es-query",
"actions": [
{
"group": "query matched",
"id": "${ACTION_ID}",
"params": {
"level": "info",
"message": """Elasticsearch query rule '{{rule.name}}' is active:
- Value: {{context.value}}
- Conditions Met: {{context.conditions}} over {{rule.params.timeWindowSize}}{{rule.params.timeWindowUnit}}
- Timestamp: {{context.date}}
- Link: {{context.link}}"""
},
"frequency": {
"notify_when": "onActionGroupChange",
"throttle": null,
"summary": false
}
}
],
"alert_delay": {
"active": 3
}
}
```
- Verify that the alert will not be created until it has matched the
delay threshold.
- Verify that the delay does not affect recovered alerts
## Summary
This PR adds a Detection Rules Counter on Rules Flyout, users are also
able to add Detection Rule on Flyout
a000c1d5-3cec-4631-b255-745547e93c38
When Rule is disabled, the text 'Disabled' will be rendered instead of
Detection Rule Counter
<img width="1423" alt="Screenshot 2024-02-01 at 8 09 04 PM"
src="aad119d4-75a3-4407-8155-fb68d7d06f38">
## Summary
follow-up of https://github.com/elastic/kibana/pull/173185
This PR enables the persistence layer implemented in the previous PR,
applying the following changes:
- Update the mapping to store unitary index results instead of storing
the whole pattern with the results in each document.
- Change the query to get the stored results by aggregating documents by
indexName. The authorized indexNames derived from the `pattern`
parameter are retrieved using the `indices.get` request.
- A bug involving a race condition with the initialization and the
retrieval of stored results, resulting in an unintended reset of the
results in the UI, has been fixed.
0598606b-c5f4-42b3-901c-f86a3cac65e4
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Additional fixes missed in https://github.com/elastic/kibana/pull/173366
- The deployment first paragraph was outdated and irrelevant to the user
workflow
- Some of step headings were vague
## Summary
Fixes the colors of "value dots", the dots that are shown in the Single
Metric Viewer when theres sparse data and not a continuous line.
Before:
<img width="1112" alt="image"
src="7844bfa0-3a50-4088-869a-5a6fc366c0cd">
After:
<img width="1112" alt="image"
src="28fcaa1e-eed9-497b-8c1a-c824effd7c31">
### Checklist
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Summary
The logstash output in fleet was not following the
[RFC-952](https://www.rfc-editor.org/rfc/rfc952) correctly, only
accepting lowercase url's for the hostname, even though according to the
standard, this should be irrelevant.
Additionally added two new unit tests, to check that this behaviour is
compliant in the future as well.
## Fix explanation
The issue was in the past that the URL Type in nodejs is automatically
lowercase, and then the value of the form is or the string provided is
checked against this url type. The types behaviour is explained here:
https://nodejs.org/api/url.html#the-whatwg-url-api
As we would like to be case insensitive, in the check, the value is also
lowerecased with: val.toLowerCase() to pass the check
### Checklist
Delete any items that are not applicable to this PR.
- [X] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
In this PR the Kibana image override is introduced.
The user will be able to decide whether the pipeline will be testing
against the code that is already deployed in QA **OR** if the project
will be created with an override from the latest commit in kibana.
This override is created in the first step of the pipeline where a new
freshly baked kibana image is built and pushed into
`docker.elastic.co/kibana-ci/kibana-serverless` with tag
`git-${BUILDKITE_COMMIT:0:12}` which is the exact commit checked out in
the current build. This is then pushed to the above mentioned registry
and as a result an image like
`docker.elastic.co/kibana-ci/kibana-serverless:git-46iuy423rn3` is being
used as the override.
In order to run the tests with the override the environment variable
`KIBANA_MKI_USE_LATEST_COMMIT=1` needs to be passed when starting the
build as shown in the following snippet.
By default, if the environment variable is not passed, the tests will
run against the already deployed version in QA without any override.
**If only** the override flow is picked, then the Build Kibana Image
will build and push the new image. On the other hand if the environment
variable is absent, then the Build Kibana Image step will exit
immediately with exit code 0 (passed in buildkite pipeline) and proceed
to run the tests normally.
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR fixes the paper cut where the URL template editor in URL
drilldown is unusably small. It now can expand as you type longer URLs
fix https://github.com/elastic/kibana/issues/132513
The input box now expands from 5 to 15 lines.
## Summary
Fixes https://github.com/elastic/kibana/issues/176227
As not everyone is using expressions and the `?drop_null_columns`
significantly changing the output, I am making this configurable. So for
the expressions users it will always be true but for anyone else it
would be false unless they specifically configure it.
This PR is fixing it but Nathan is going to follow up:
- maps are setting the drop_null_columns to true (as it makes the
response smaller, which is always a benefit)
- add some FTs to test the output
---------
Co-authored-by: Julia Rechkunova <julia.rechkunova@gmail.com>
## Summary
Closes https://github.com/elastic/kibana/issues/161194
Dashboard with Lens, aggbased and TSVB using the legacy complimentary
palette.
<img width="2104" alt="image"
src="9cc0da68-3f36-401a-a264-a15e71c46cdd">
Correct usage in the list.
<img width="530" alt="image"
src="4f9046f8-77ae-4d30-b12f-92844d77a758">
I don't want to run a migration. All the new visualizations will have
the correct id in the SO and I am fallbacking for the existing. As with
the new color mapping we are going to construct this palette (it won't
be in the list) I think that this is enough to solve the problem.
## Summary
Follow up on #174585
Less code, less iterations.
---------
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
Co-authored-by: Stratoula Kalafateli <stratoula1@gmail.com>
Co-authored-by: Julia Rechkunova <julia.rechkunova@gmail.com>
Closes#171747Fixes#174285Fixes#174286Fixes#174287Fixes#174288Fixes#174289
## Summary
This feature branch introduces default values to the custom fields in
Cases.
Changes:
- The case configuration API now supports defining default values **for
required custom fields**.
- The default value field is optional.
- The default value field must have the same type as the custom field.
- Default values for required custom fields can be configured in the
cases settings page.
- The Create and Update Case pages now prefill required custom fields
with their default value(if it exists).
- Create and Update case API
- required custom field missing from the request:
- if the default value exists -> set the custom field to the default
value
- if the default value is missing from the configuration -> throws an
error
- creating required custom fields with `value: null` -> always throws an
error
### Previous PRs
#174043#174845#174628#175197#175978#175795
## Testing
I'll leave here a couple of important use cases to test:
- Migration scenario
1. Create **multiple** required custom fields in 8.12.
2. Migrate to the feature branch (future 8.13).
3. Defining default values in the settings page should work as expected.
- Update case UI
1. Create an optional custom field
2. Create a case and leave the custom field empty
3. Edit the custom field to now be required
4. Go to the case detail page and edit the custom field
5. Is the form populated automatically with the default value?
## Release notes
Required custom fields now support default values. These default values
will be used to populate automatically the custom fields if they are not
defined when creating and updating cases.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
- Use Connectors API endpoint `_connector/{id}/_index_name` to update
connector `index_name`. Index name validation and usage (if it's used by
another connector already) is done on backend in ES.
- Use Connectors API endpoint `_connector/{id}/_status` to update
connector status.
## Summary
Currently the Help menu for serverless points to the [Elastic
documentation page](https://www.elastic.co/guide/index.html), which
shows all the Elastic documentation and defaults to the Solutions tab.
It would be very easy for serverless users to follow the Help link and
end up in the stateful docs instead of serverless.
This PR overrides the URL in the Help menu to send users directly to the
[serverless docs](https://docs.elastic.co/serverless).
### Checklist
n/a
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Resolves https://github.com/elastic/kibana/issues/176116
## Summary
The test was failing with the following error, `Incorrect HTTP method
for uri [/_esql] and method [POST], allowed: [HEAD, PUT, DELETE,
GET]","status":405`.
This pr updates the ES|QL query request, for es query rule, to use`
/_query`. Security solution also uses `/_query` for their ES|QL rule
here:
`x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/esql/esql_request.ts`.
### To verify
- Run this command to start the functional test server, and verify that
the tests work properly
```
ES_SNAPSHOT_MANIFEST="20240201-212246_98a37c7b/manifest.json" node scripts/functional_tests_server.js --config x-pack/test/alerting_api_integration/spaces_only/tests/alerting/group3/config.ts
```
In this PR, I'm adding a test in the alerting framework to detect
changes in a rule type's params schema that will require a snapshot to
be updated. This snapshot will provide a centralized place to view
history on alerting rule params in case we need to asses risk for
rolling upgrades or rollbacks of a release (serverless). The only rule
types affected are those running in serverless in any of the three
project types.
When a rule type is used in serverless, it must provide one of the
following configuration to their rule type on top of everything else:
```
// Zod schema
schemas: {
params: {
type: 'zod',
schema: UnifiedQueryRuleParams
},
},
// config-schema
schemas: {
params: {
type: 'config-schema',
schema: EsQueryRuleParamsSchema,
},
},
```
We are working on documenting guidelines so engineers and response ops
can ensure a change to rule parameters will work properly in rolling
upgrade and rollback scenarios and be part of the PR review process.
NOTE to rule type owners: I pass the same schema used to validate to the
`schemas.params` attribute in the rule type. It will be important to
keep them in sync. Down the road, we plan to make `validate.params`
optional and use the schema as a starting point so it's easier to have a
single variable passed in.
## To verify
1. Make changes to the params schema of the ES query rule type.
```
diff --git a/x-pack/plugins/stack_alerts/server/rule_types/es_query/rule_type_params.ts b/x-pack/plugins/stack_alerts/server/rule_types/es_query/rule_type_params.ts
index 73e8eae32cf..09ec74104ec 100644
--- a/x-pack/plugins/stack_alerts/server/rule_types/es_query/rule_type_params.ts
+++ b/x-pack/plugins/stack_alerts/server/rule_types/es_query/rule_type_params.ts
@@ -39,6 +39,7 @@ export type EsQueryRuleParamsExtractedParams = Omit<EsQueryRuleParams, 'searchCo
};
const EsQueryRuleParamsSchemaProperties = {
+ foo: schema.boolean(),
size: schema.number({ min: 0, max: ES_QUERY_MAX_HITS_PER_EXECUTION }),
timeWindowSize: schema.number({ min: 1 }),
excludeHitsFromPreviousRun: schema.boolean({ defaultValue: true }),
```
2. Run the jest integration test to update the snapshot file
```
node scripts/jest_integration.js x-pack/plugins/alerting/server/integration_tests/serverless_upgrade_and_rollback_checks.test.ts -u
```
3. Notice the
`x-pack/plugins/alerting/server/integration_tests/__snapshots__/serverless_upgrade_and_rollback_checks.test.ts.snap`
file got updated
```
"foo": Object {
"flags": Object {
"error": [Function],
},
"type": "boolean",
},
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
