## Summary
Adds new API for getting uninstall tokens either for all policies or for
a selected policy:
- endpoint: `GET /api/fleet/uninstall_tokens`
- optional query params: `page`, `perPage` and `policyId`
- **update**: paginates in the `UninstallTokenService`, using the SO
query
(before: ~paginates result: this is a possibly temporary pagination
solution implemented in the handler, as the service is complex enough
and expected to change around the aggregation -> therefore API
integration tests enforce the contract for pagination for easier
modification in the future instead of unit tests~)
- **update**: responses with a `ListResult<UninstallToken>` type
(before: ~returns a `map` of the uninstall tokens: `{ [policyId]: {
token, created_at } }`, with the usual `total`, `perPage` and `page`
fields~)
- also added (a hopefully correct) `openAPI` definition
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Part of https://github.com/elastic/kibana/issues/154731
Allow secrets to be updated via the API. When a secret value is updated,
the secret reference is replaced with a "raw" value we detect this on
the API and create a new secret document.
Once a secret reference is updated, we clean up the old secret document
if it is not in use by another policy. This check is a simple lookup of
the secret_references array on policies.
API integration tests updated.
## Summary
The first part of secrets phase 1. This is not a fully working
implementation of secrets just yet, hence why it is behind a feature
flag. This just implements creating secrets:
- on package policy creation, if a package has fields with `secrets:
true` set, then their values are stored in the .secrets system index, a
reference to the secret is stored on the package policy e.g { id : 1234
isSecretReference : true }
- The compiled policy (returned from the get full agent policy API, or
stored in the .fleet-policies index) shows secret values in the format
`$co.elastic.secret{12345}` and includes a top level secret_references
array with all secret IDs in it, allowing fleet server to look them up
in one swoop.
- This works for pakacge level vars, input level vars and stream level
vars
Part of https://github.com/elastic/kibana/issues/154715
How to test:
```
# clone the elasticsearch repo
gh pr checkout 95625
./gradlew run
# now get a service token
curl -XPOST -u elastic:password http://localhost:9200/_security/service/elastic/kibana/credential/token/token1
# paste the service token into your kibana config under
# elasticsearch.serviceAccountToken: "<your_token>"
# once kibana has started, we now need to run our own package registry to get a package with secrets in
# replace /Users/markhopkin/dev with the path to your kibana
docker run -p 8080:8080 -v /Users/markhopkin/dev/kibana/x-pack/test/fleet_api_integration/apis/fixtures/test_packages:/packages/test-packages -v /Users/markhopkin/dev/kibana/x-pack/test/fleet_api_integration/apis/fixtures/package_registry_config.yml:/package-registry/config.yml docker.elastic.co/package-registry/package-registry:main
# once kibana has started successfully once and installed the fleet_server package, add this to your kibana config
xpack.fleet.registryUrl: http://localhost:8080
# you can now see the 'secrets' package and create a package policy
# after creating the package policy, check the .fleet-secrets index, the .fleet-policies index or the get package policy API to see how the secrets have been stored
```
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Runtime for `fleet_api_integration/config` crossed the 38 min limit.
Ideally config runtime should be <10 min so that pipeline can quickly
retry failed configs (we do 1 retry on CI)
<img width="1600" alt="image"
src="https://user-images.githubusercontent.com/10977896/235707744-3120e1e9-4882-493f-9ee0-86016a765401.png">
This PR splits the existing config into few smaller ones:
```
x-pack/test/fleet_api_integration/config.agent.ts 15m 15s
x-pack/test/fleet_api_integration/config.agent_policy.ts 4m 10s
x-pack/test/fleet_api_integration/config.epm.ts 8m 12s
x-pack/test/fleet_api_integration/config.package_policy.ts 10m 54s
// combines multiple test files
x-pack/test/fleet_api_integration/config.fleet.ts 5m 2s
```
## Summary
Closes https://github.com/elastic/kibana/issues/145529
For integration packages, index templates are created at install time
because the package contains all information needed to create the data
stream. Input packages need to create the index templates at package
policy creation time so that dataset can be populated.
Summary of changes:
- when creating a package policy for an input package, the correct index
templates and ingest pipelines are created, for example for the dataset
`dataset1` the following will be created (and added to `installed_es` on
the installation saved object):
- `logs-dataset1-1.0.0` (ingest_pipeline)
- `logs-dataset1` (index_template)
- `logs-dataset1@package` (component template)
- `logs-dataset1@custom'`(component template)
- when a dataset matches an existing data stream
- if the existing data stream is from the same package, do not create
any new index templates as existing ones will be used
- if the existing data stream is from a different package, the API will
reject the request unless the force flag is used.
- when upgrading an input package, all dynamically created assets will
be updated as well.
- when uninstalling an input package, all dynamically created assets
will be uninstalled
- bonus: support the new top level `elasticsearch` field for input
package manifests (needed this field for upgrade testing)
### Test setup
To test we need a docker registry with input packages, the easiest way
is to use the test fixtures from the kibana repo (replace directory with
your own)
```
docker run -p 8080:8080 -v /Users/markhopkin/dev/kibana/x-pack/test/fleet_api_integration/apis/fixtures/test_packages:/packages/test-packages -v /Users/markhopkin/dev/kibana/x-pack/test/fleet_api_integration/apis/fixtures/package_registry_config.yml:/package-registry/config.yml docker.elastic.co/package-registry/package-registry:main
```
And add this to your kibana yml config:
```
xpack.fleet.registryUrl: http://localhost:8080
```
this will make the test package `input_package_upgrade` available which
is a version of the custom logs integration:
`http://<your_kibana>/app/integrations/detail/input_package_upgrade-1.0.0/overview`
### Test scenarios
#### 1. Package policy creation (new datastream)
- with input_package_upgrade version 1.0.0 installed and an agent policy
with at least one agent
- create a package policy with a valid logfile and `dataset1` as the
dataset
- logs-dataset1 index template should have been created
- add an agent to the package policy
- append to the log file
- data should be added to the logs-dataset1-default datastream
##### 2. Package policy creation (existing datastream same package)
- with input_package_upgrade version 1.0.0 installed and an agent policy
with at least one agent
- create **another** package policy with a valid logfile and `dataset1`
as the dataset
- logs-dataset1 should still exist
- append to the log file
- data should be added to the logs-dataset1-default datastream
##### 3. Package policy creation (existing datastream different package)
- with input_package_upgrade version 1.0.0 installed and an agent policy
with at least one agent
- ensure there are some other fleet data streams on the system (i.e data
has been ingested), e.g logs-elastic-agent
- create a package policy with a valid logfile and `elastic-agent` as
the dataset
-the package policy should be successfully created
- append to the log file
- data should be added to the logs-elastic-agent-default datastream
##### 3b. Package policy creation (existing index template different
package)
- with input_package_upgrade version 1.0.0 installed and an agent policy
with at least one agent
- ensure there is another fleet index template on the system with no
matching data streams (i.e no data has been ingested), e.g
logs-system.auth from the system package
- create a package policy with a valid logfile and `system.auth` as the
dataset
-the package policy should be successfully created
- append to the log file
- data should be added to the logs-system.auth-default datastream
- the `logs-system.auth` index template should still have`
_meta.package.name` set to 'system'
<img width="650" alt="Screenshot 2023-01-17 at 21 31 10"
src="https://user-images.githubusercontent.com/3315046/213016570-daab98e4-9cc2-479a-9349-9fd727f9d899.png">
##### 4. Package policy delete
- with input_package_upgrade version 1.0.0 installed and an agent policy
with at least one agent
- ensure there are some other fleet data streams on the system, e.g
logs-elastic-agent
- create one or many package policys with a valid logfile and different
datasets
- note all of the index templates created
- uninstall the package
- all created index templates should be deleted
##### 5. package policy upgrade
- with input_package_upgrade version 1.0.0 installed and an agent policy
with at least one agent
- create one or many package policys with a valid logfile and different
datasets
- note all of the index templates created
- upgrade to input_package_upgrade version 1.1.0, this adds
`mappings.properties.@timestamp` to the `@package` component template
for all data streams:
```
mappings:
properties:
'@timestamp':
ignore_malformed: false
type: date
```
- verify all new data streams have the new property
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
- [x] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [x] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
* [Fleet] Create new API to manage download_source setting
* Add preconfiguration and integration tests
* Add more tests
* Update openapi docs to cover new api
* Fix broken tests and typo
* Fix failing tests
* Check that host is a valid uri
* Fix incorrect method call
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [ftr] automatically determine config run order
* split lens config into two groups
* support ftr configs always running against CI
* Split detection_engine_api_integration rule exception list tests
* Add configs from previous commit
* [ftr] remove testMetadata and maintain a unique lifecycle instance per run
* Revert "[ftr] remove testMetadata and maintain a unique lifecycle instance per run"
This reverts commit d2b4fdb824.
* Split alerting_api_integration/security_and_spaces tests
* Add groups to yaml
* Revert "Revert "[ftr] remove testMetadata and maintain a unique lifecycle instance per run""
This reverts commit 56232eea68.
* stop ES more forcefully and fix timeout
* only cleanup lifecycle phases when the cleanup is totally complete
* only use kill when cleaning up an esTestInstance
* fix broken import
* fix runOptions.alwaysUseSource implementation
* fix config access
* fix x-pack/ccs config
* fix ml import file paths
* update kibana build id
* revert array.concat() change
* fix baseConfig usage
* fix pie chart data
* split up maps tests
* pull in all of group5 so that es archives are loaded correctly
* add to ftr configs.yml
* fix pie chart data without breaking legacy version
* fix more pie_chart stuff in new vis lib
* restore normal PR tasks
* bump kibana-buildkite-library
* remove ciGroup validation
* remove the script which is no longer called from checks.sh
* [CI] Auto-commit changed files from 'yarn kbn run build -i @kbn/pm'
* adapt flaky test runner scripts to handle ftrConfig paths
* fix types in alerting_api_integration
* improve flaky config parsing and use non-local var name for passing explicit configs to ftr_configs.sh
* Split xpack dashboard tests
* Add configs
* [flaky] remove key from ftr-config steps
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
* restore cypress builds
* remove ciGroups from FTR config files
* fixup some docs
* add temporary script to hunt for FTR config files
* use config.base.js naming for clarity
* use script to power ftr_configs.yml
* remove usage of removed x-pack/scripts/functional_tests
* fix test names in dashboard snapshots
* bump kibana-buildkite-library
* Try retrying only failed configs
* be a little quieter about trying to get testStats from configs with testRunners defined
* Remove test code
* bump kibana-buildkite-library
* update es_snapshot and on_merge jobs too
* track duration and exit code for each config and print it at the end of the script
* store results in order, rather than by key, in case there are duplicates in $config
* bash is hard
* fix env source and use +e rather than disabling e for whole file
* bash sucks
* print config summary in jest jobs too
* define results in jest_parallel.sh
* simplify config summary print, format times a little better
* fix reference to unbound time variable, use better variable name
* skip the newline between each result
* finish with the nitpicking
* sync changes with ftr_configs.sh
* refuse to execute config files which aren't listed in the .buildkite/ftr_configs.yml
* fix config.edge.js base config import paths
* fix some readmes
* resolve paths from ftr_configs manifest
* fix readConfigFile tests
* just allow __fixtures__ configs
* list a few more cypress config files
* install the main branch of kibana-buildkite-library
* split up lens group1
* move ml data_visualizer tests to their own config
* fix import paths
* fix more imports
* install specific commit of buildkite-pipeline-library
* sort configs in ftr_configs.yml
* bump kibana-buildkite-library
* remove temporary script
* fix env var for limiting config types
* Update docs/developer/contributing/development-functional-tests.asciidoc
Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
* produce a JUnit report for saved objects field count
* apply standard concurrency limits from flaky test runner
* support customizing FTR concurrency via the env
Co-authored-by: Brian Seeders <brian.seeders@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
* Add Integrations page callout for package upgades
* Fix props
* Add missing file
* Add integrations upgrade callout message
* Add link to updates available tab
* Fix merge
* Upgrade ppolicies UI WIP
* Initial upgrade dry run API
* Add upgrade method
* Move overridePackageInputs and use for upgrade method
* Add new variables to dry run diff
* Revert UI changes to uto upgrade wizard
* Add vars and streams to error keys
* Type fix
* Fix jest
* Fix types
* Fix typecheck
* Fix types
* Add integration test for dry run API
* Flesh out test cases
* Clean up error responses for dry runs
* Fix failing tests
* WIP: Add (failing for now) test case for package upgrade w/ error
* Add compiled_stream to test API payload
* Fix failing test case for automatic upgrade
* Fix compiled stream in package policy upgrade
* Remove fleet and agent setup from integration test
* Unload esarchiver fixtures in api integration test
Co-authored-by: Kyle Pollich <kpollich1@gmail.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Don't block standalone agent instructions when not using Fleet server yet
* Add service token instructions - UI only
* Add route for regenerating fleet server service token
* generate tokens instead of regenerate and add error catching and tests
* fix i18n typo
* i18n fix, add sudo, copy edits
* Fix commands
* Add missing test file
Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Add force option to DELETE package endpoint.
* Add integration test.
* Adjust openapi spec.
* Run EPM tests before fleet setup tests.
* Run package delete tests first in EPM tests
* Add API integration tests for data streams list, including one that is expected to fail due to reliance on number of backing indices
* Use ES data streams API as source of truth for list of data streams, and only query against backing indices afterwards
* Get package name from data stream meta info
* Increate retry timeout
* Move initial info requests inside Promise.all
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
