Commit graph

86 commits

Author SHA1 Message Date
Kibana Machine
8ea7cee500
[8.10] [esArchiver] Do not perform SO migration on cleanup (#163302) (#165451)
# Backport

This will backport the following commits from `main` to `8.10`:
- [[esArchiver] Do not perform SO migration on cleanup
(#163302)](https://github.com/elastic/kibana/pull/163302)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Gerard
Soldevila","email":"gerard.soldevila@elastic.co"},"sourceCommit":{"committedDate":"2023-09-01T10:41:20Z","message":"[esArchiver]
Do not perform SO migration on cleanup (#163302)\n\n##
Summary\r\n\r\nAttempt at fixing
https://github.com/elastic/kibana/issues/127545\r\n\r\nThe failing tests
use `esArchiver` to cleanup Saved Object indices\r\nbetween
tests.\r\nStrangely, the cleanup method calls the migration logic to
update the SO\r\nindices.\r\n\r\nIn this particular source of flakiness,
a test fails during the\r\nmigration (problem is tracked
by\r\nhttps://github.com/elastic/kibana/issues/163289).\r\n\r\nPerforming
a migration as part of the cleanup does not make too much\r\nsense (at
least in this test suite), so the goal of this PR is to\r\nsimplify the
cleanup operation, getting rid of the call
to\r\n`migrateSavedObjectIndices`.\r\n\r\n50 runs flaky test
runner\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2850\r\n","sha":"f8303ec7cf57aa8787331ada397dfa853de969c9","branchLabelMapping":{"^v8.11.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","Team:Core","Team:Operations","release_note:skip","test-failure-flaky","backport:all-open","v8.11.0"],"number":163302,"url":"https://github.com/elastic/kibana/pull/163302","mergeCommit":{"message":"[esArchiver]
Do not perform SO migration on cleanup (#163302)\n\n##
Summary\r\n\r\nAttempt at fixing
https://github.com/elastic/kibana/issues/127545\r\n\r\nThe failing tests
use `esArchiver` to cleanup Saved Object indices\r\nbetween
tests.\r\nStrangely, the cleanup method calls the migration logic to
update the SO\r\nindices.\r\n\r\nIn this particular source of flakiness,
a test fails during the\r\nmigration (problem is tracked
by\r\nhttps://github.com/elastic/kibana/issues/163289).\r\n\r\nPerforming
a migration as part of the cleanup does not make too much\r\nsense (at
least in this test suite), so the goal of this PR is to\r\nsimplify the
cleanup operation, getting rid of the call
to\r\n`migrateSavedObjectIndices`.\r\n\r\n50 runs flaky test
runner\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2850\r\n","sha":"f8303ec7cf57aa8787331ada397dfa853de969c9"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.11.0","labelRegex":"^v8.11.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/163302","number":163302,"mergeCommit":{"message":"[esArchiver]
Do not perform SO migration on cleanup (#163302)\n\n##
Summary\r\n\r\nAttempt at fixing
https://github.com/elastic/kibana/issues/127545\r\n\r\nThe failing tests
use `esArchiver` to cleanup Saved Object indices\r\nbetween
tests.\r\nStrangely, the cleanup method calls the migration logic to
update the SO\r\nindices.\r\n\r\nIn this particular source of flakiness,
a test fails during the\r\nmigration (problem is tracked
by\r\nhttps://github.com/elastic/kibana/issues/163289).\r\n\r\nPerforming
a migration as part of the cleanup does not make too much\r\nsense (at
least in this test suite), so the goal of this PR is to\r\nsimplify the
cleanup operation, getting rid of the call
to\r\n`migrateSavedObjectIndices`.\r\n\r\n50 runs flaky test
runner\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2850\r\n","sha":"f8303ec7cf57aa8787331ada397dfa853de969c9"}}]}]
BACKPORT-->

Co-authored-by: Gerard Soldevila <gerard.soldevila@elastic.co>
2023-09-01 05:02:39 -07:00
Anton Dosov
c76b185323
[uiSettings] use refresh: false instead of default refresh: 'wait_for' in create and update (#160278)
## Summary

Close https://github.com/elastic/kibana/issues/159662

See explanation [here](https://github.com/elastic/kibana/issues/159662)
2023-07-25 04:21:28 -07:00
Gerard Soldevila
2e15549b58
[FTR - esArchiver] Update the remaining "dynamic" archives, removing mappings.json (#162029)
## Summary

Part of https://github.com/elastic/kibana/issues/161882

- Updates the remaining "dynamic" es archives (aka those using the
`$KIBANA_PACKAGE_VERSION` variable), deleting mappings.json.
- It also removes the piece of logic in `esArchiver` that replaces that
variable.
2023-07-19 17:43:00 +02:00
Gerard Soldevila
79f7bb45fd
Attempt to fix flaky tests after split migration (#159397)
## Summary
Addresses root cause of https://github.com/elastic/kibana/issues/158918

Underlying cause is that _esArchiver_ is messing up with the SO indices
whilst Kibana is already running.

This can cause some asynchronous calls made by Kibana (e.g. `GET
/.kibana_8.8.0/telemetry:telemetry`) to hit ES at the exact time where
the underlying SO indices are **just** recreated, causing the error
described in the related issue.

The idea of the fix is to delete `mappings.json`, used by _esArchiver_
to create the SO indices. This way, _esArchiver_ will use existing SO
indices instead (aka the "official" ones, created by Kibana at startup),
thus avoiding the problem altogether.

As a side effect:

- Documents in `data.json` must be updated so that they are correctly
inserted.
- The different FTR tests must make sure the SO indices are empty before
inserting those documents (done in the `before(), beforeEach()`
statements).
2023-07-07 17:05:42 +02:00
Gerard Soldevila
295b4d4a34
[esArchiver] Update aliases after creating the indices (#160584)
Tackles https://github.com/elastic/kibana/issues/158918

Updates `esArchiver` so that SO indices are created in two separate
steps:

`indices.create()` and `indices.updateAliases()`

This way, any Kibana requests that target SO indices (through their
aliases) will either find that the indices exist, or that they do not.

This is a less invasive approach than
https://github.com/elastic/kibana/pull/159397, as it does not modify the
`esArchiver.load` flow (we don't delete the `mappings.json` files here).
2023-06-29 11:30:15 +02:00
Gerard Soldevila
bbb5fc4abe
[esArchiver] Automatically cleanup SO indices when SO documents are found in data.json (#159582)
The ultimate goal of this PR is to lay the groundwork to be able to
remove the "dynamic" `mappings.json`, which probably should have never
existed.

With the PR, detecting SO documents in the `data.json` will
automatically trigger a cleanup of the SO indices.
This, in turn, will allow not having to define "dynamic" saved objects
indices (i.e. those with the `$KIBANA_PACKAGE_VERSION` variable in the
`mappings.json`).

IIUC the idea behind the dynamic indices was to have SO indices that are
aligned with the current stack version, avoiding the extra overhead of
having to migrate the inserted documents, and reducing overall test
times.

Nonetheless, what is happening today is:
1. FTR starts ES and Kibana.
2. Kibana creates current version SO indices at startup (empty ones).
3. `esArchiver.load()` processes the `mappings.json`.
3.1. It detects that we are defining SO indices and **deletes** existing
saved object indices.
3.2 It then re-creates these indices according to the definitions on
`mappings.json`.
4. `esArchiver.load()` processes the `data.json`. Specifically, it
inserts SO documents present in `data.json`.
5. `esArchiver.load()` calls the _KibanaMigrator_ to make sure that the
inserted documents are up-to-date, hoping they are already aligned with
current stack version (which is not always the case, not even with
"dynamic" mappings).

Two interesting things to note:
- Steps 3 to 5 happen whilst Kibana is already started and running. If
Kibana queries SO indices during `esArchiver.load()`, and a request to
ES is made **right after** 3.2, the result might be
https://github.com/elastic/kibana/issues/158918.
- Having dynamic SO indices' definitions, deleting the "official"
indices created by Kibana (3.1), and recreating them hoping to be
aligned with current stack version (3.2) is non-sense. We could use the
existing SO indices instead, and simply clean them up whenever we are
about to insert SO documents.

Performing that cleanup is precisely the goal of this PR.
Then, in subsequent PRs like
https://github.com/elastic/kibana/pull/159397/files, tackling the flaky
tests, we'll be able to simply remove the "dynamic" `mappings.json`
definitions, causing `esArchiver` to rely on SO indices created by
Kibana.

Thanks to this PR, the FTR tests won't need to explicitly cleanup saved
object indices in the `before` hooks.
2023-06-19 13:08:03 +02:00
Dzmitry Lemechko
7bbdab0653
[codeowners] add appex-qa for ftr-related packages (#155230)
## Summary

While elastic/appex-qa is in process to take ownership over FTR related
code, from elastic/kibana-operations it will be good to keep track on
any changes happening and participate in the review process.
2023-05-24 08:53:09 +02:00
Gerard Soldevila
21351df953
Split the .kibana saved objects index into multiple indices (#154888)
## Description 

Fix https://github.com/elastic/kibana/issues/104081

This PR move some of the SO types from the `.kibana` index into the
following ones:
- `.kibana_alerting_cases`
- `.kibana_analytics`
- `.kibana_security_solution`
- `.kibana_ingest`

This split/reallocation will occur during the `8.8.0` Kibana upgrade
(*meaning: from any version older than `8.8.0` to any version greater or
equal to `8.8.0`*)

**This PR main changes are:**
- implement the changes required in the SO migration algorithm to
support this reallocation
- update the FTR tools (looking at you esArchiver) to support these new
indices
- update hardcoded references to `.kibana` and usage of the
`core.savedObjects.getKibanaIndex()` to use new APIs to target the
correct index/indices
- update FTR datasets, tests and utility accordingly 

## To reviewers

**Overall estimated risk of regressions: low**

But, still, please take the time to review changes in your code. The
parts of the production code that were the most impacted are the
telemetry collectors, as most of them were performing direct requests
against the `.kibana` index, so we had to adapt them. Most other
contributor-owned changes are in FTR tests and datasets.

If you think a type is misplaced (either we missed some types that
should be moved to a specific index, or some types were moved and
shouldn't have been) please tell us, and we'll fix the reallocation
either in this PR or in a follow-up.

## .Kibana split

The following new indices are introduced by this PR, with the following
SO types being moved to it. (any SO type not listed here will be staying
in its current index)

Note: The complete **_type => index_** breakdown is available in [this
spreadsheet](https://docs.google.com/spreadsheets/d/1b_MG_E_aBksZ4Vkd9cVayij1oBpdhvH4XC8NVlChiio/edit#gid=145920788).

#### `.kibana_alerting_cases`
- action
- action_task_params
- alert
- api_key_pending_invalidation
- cases
- cases-comments
- cases-configure
- cases-connector-mappings
- cases-telemetry
- cases-user-actions
- connector_token
- rules-settings
- maintenance-window

#### `.kibana_security_solution`
- csp-rule-template
- endpoint:user-artifact
- endpoint:user-artifact-manifest
- exception-list
- exception-list-agnostic
- osquery-manager-usage-metric
- osquery-pack
- osquery-pack-asset
- osquery-saved-query
- security-rule
- security-solution-signals-migration
- siem-detection-engine-rule-actions
- siem-ui-timeline
- siem-ui-timeline-note
- siem-ui-timeline-pinned-event

#### `.kibana_analytics`

- canvas-element
- canvas-workpad-template
- canvas-workpad
- dashboard
- graph-workspace
- index-pattern
- kql-telemetry
- lens
- lens-ui-telemetry
- map
- search
- search-session
- search-telemetry
- visualization

#### `.kibana_ingest`

- epm-packages
- epm-packages-assets
- fleet-fleet-server-host
- fleet-message-signing-keys
- fleet-preconfiguration-deletion-record
- fleet-proxy
- ingest_manager_settings
- ingest-agent-policies
- ingest-download-sources
- ingest-outputs
- ingest-package-policies

## Tasks / PRs

### Sub-PRs

**Implementation**
- 🟣 https://github.com/elastic/kibana/pull/154846
- 🟣 https://github.com/elastic/kibana/pull/154892
- 🟣 https://github.com/elastic/kibana/pull/154882
- 🟣 https://github.com/elastic/kibana/pull/154884
- 🟣 https://github.com/elastic/kibana/pull/155155

**Individual index split**
- 🟣 https://github.com/elastic/kibana/pull/154897
- 🟣 https://github.com/elastic/kibana/pull/155129
- 🟣 https://github.com/elastic/kibana/pull/155140
- 🟣 https://github.com/elastic/kibana/pull/155130

### Improvements / follow-ups 

- 👷🏼 Extract logic into
[runV2Migration](https://github.com/elastic/kibana/pull/154151#discussion_r1158470566)
@gsoldevila
- Make `getCurrentIndexTypesMap` resillient to intermittent failures
https://github.com/elastic/kibana/pull/154151#discussion_r1169289717
- 🚧 Build a more structured
[MigratorSynchronizer](https://github.com/elastic/kibana/pull/154151#discussion_r1158469918)
- 🟣 https://github.com/elastic/kibana/pull/155035
- 🟣 https://github.com/elastic/kibana/pull/155116
- 🟣 https://github.com/elastic/kibana/pull/155366
## Reallocation tweaks

Tweaks to the reallocation can be done after the initial merge, as long
as it's done before the public release of 8.8

- `url` should get back to `.kibana` (see
[comment](https://github.com/elastic/kibana/pull/154888#discussion_r1172317133))

## Release Note

For performance purposes, Kibana is now using more system indices to
store its internal data.

The following system indices will be created when upgrading to `8.8.0`:

- `.kibana_alerting_cases`
- `.kibana_analytics`
- `.kibana_security_solution`
- `.kibana_ingest`

---------

Co-authored-by: pgayvallet <pierre.gayvallet@elastic.co>
Co-authored-by: Christos Nasikas <christos.nasikas@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>
2023-04-25 09:43:42 +02:00
Spencer
afb09ccf8a
Transpile packages on demand, validate all TS projects (#146212)
## Dearest Reviewers 👋 

I've been working on this branch with @mistic and @tylersmalley and
we're really confident in these changes. Additionally, this changes code
in nearly every package in the repo so we don't plan to wait for reviews
to get in before merging this. If you'd like to have a concern
addressed, please feel free to leave a review, but assuming that nobody
raises a blocker in the next 24 hours we plan to merge this EOD pacific
tomorrow, 12/22.

We'll be paying close attention to any issues this causes after merging
and work on getting those fixed ASAP. 🚀

---

The operations team is not confident that we'll have the time to achieve
what we originally set out to accomplish by moving to Bazel with the
time and resources we have available. We have also bought ourselves some
headroom with improvements to babel-register, optimizer caching, and
typescript project structure.

In order to make sure we deliver packages as quickly as possible (many
teams really want them), with a usable and familiar developer
experience, this PR removes Bazel for building packages in favor of
using the same JIT transpilation we use for plugins.

Additionally, packages now use `kbn_references` (again, just copying the
dx from plugins to packages).

Because of the complex relationships between packages/plugins and in
order to prepare ourselves for automatic dependency detection tools we
plan to use in the future, this PR also introduces a "TS Project Linter"
which will validate that every tsconfig.json file meets a few
requirements:

1. the chain of base config files extended by each config includes
`tsconfig.base.json` and not `tsconfig.json`
1. the `include` config is used, and not `files`
2. the `exclude` config includes `target/**/*`
3. the `outDir` compiler option is specified as `target/types`
1. none of these compiler options are specified: `declaration`,
`declarationMap`, `emitDeclarationOnly`, `skipLibCheck`, `target`,
`paths`

4. all references to other packages/plugins use their pkg id, ie:
	
	```js
    // valid
    {
      "kbn_references": ["@kbn/core"]
    }
    // not valid
    {
      "kbn_references": [{ "path": "../../../src/core/tsconfig.json" }]
    }
    ```

5. only packages/plugins which are imported somewhere in the ts code are
listed in `kbn_references`

This linter is not only validating all of the tsconfig.json files, but
it also will fix these config files to deal with just about any
violation that can be produced. Just run `node scripts/ts_project_linter
--fix` locally to apply these fixes, or let CI take care of
automatically fixing things and pushing the changes to your PR.

> **Example:** [`64e93e5`
(#146212)](64e93e5806)
When I merged main into my PR it included a change which removed the
`@kbn/core-injected-metadata-browser` package. After resolving the
conflicts I missed a few tsconfig files which included references to the
now removed package. The TS Project Linter identified that these
references were removed from the code and pushed a change to the PR to
remove them from the tsconfig.json files.

## No bazel? Does that mean no packages??
Nope! We're still doing packages but we're pretty sure now that we won't
be using Bazel to accomplish the 'distributed caching' and 'change-based
tasks' portions of the packages project.

This PR actually makes packages much easier to work with and will be
followed up with the bundling benefits described by the original
packages RFC. Then we'll work on documentation and advocacy for using
packages for any and all new code.

We're pretty confident that implementing distributed caching and
change-based tasks will be necessary in the future, but because of
recent improvements in the repo we think we can live without them for
**at least** a year.

## Wait, there are still BUILD.bazel files in the repo
Yes, there are still three webpack bundles which are built by Bazel: the
`@kbn/ui-shared-deps-npm` DLL, `@kbn/ui-shared-deps-src` externals, and
the `@kbn/monaco` workers. These three webpack bundles are still created
during bootstrap and remotely cached using bazel. The next phase of this
project is to figure out how to get the package bundling features
described in the RFC with the current optimizer, and we expect these
bundles to go away then. Until then any package that is used in those
three bundles still needs to have a BUILD.bazel file so that they can be
referenced by the remaining webpack builds.

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2022-12-22 19:00:29 -06:00
Tiago Costa
e41569b4a6
fix(NA): wrongly spread stripInternal and rootDir configs across packages (#144463)
* chore(NA): remove overrides for rootDir on packages

* chore(NA): replace './target_types' with 'target_types' on packages

* chore(NA): removes stripInternal false configs

* chore(NA): remove unused strip internals
2022-11-03 01:04:55 +00:00
spalger
52f2b33a07
[auto] migrate existing plugin/package configs 2022-10-28 14:06:46 -05:00
spalger
42879f7656
[bazel] fix some BUILD.bazel file inconsistencies 2022-10-26 11:07:55 -05:00
Spencer
32491462a9
add kibana.jsonc files to existing packages (#138965)
* [packages] add kibana.jsonc files

* auto-migrate to kibana.jsonc

* support interactive pkg id selection too

* remove old codeowners entry

* skip codeowners generation when .github/CODEOWNERS doesn't exist

* fall back to format validation if user is offline

* update question style

* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2022-09-08 13:31:57 -07:00
Spencer
e620280d87
[docs/ops] write docs about flaky tests (#139866)
Co-authored-by: Jonathan Budzenski <jon@budzenski.me>
2022-09-01 16:23:49 +01:00
Tiago Costa
1cbf83f73c
chore(NA): remove src folder requirement from packages (part 2) (#138476)
* refact(NA): apply root_input_dir=src to each already created pkg

* refact(NA): update package generator

* fix(NA): correctly use rootDir

* fix(NA): use root input dir on latest introduced pkgs for jsts_transpiler macro

* chore(NA): merge with main

* chore(NA): first attempt to complete removal of src folder on a small group of pkgs

* Revert "chore(NA): first attempt to complete removal of src folder on a small group of pkgs"

This reverts commit b6f34b7530.

* chore(NA): remove src folder requirement from xpack pkgs

* chore(NA): remove src folder from analytics pkgs

* chore(NA): remove src folder from home pkgs

* chore(NA): remove src folder from shared_ux pkgs

* fix(NA): remove missing src folder inputs on sharedux pkg

* chore(NA): remove src folder from kbn-a* pkgs

* chore(NA): remove src folder from kbn-b* pkgs

* chore(NA): remove src folder from kbn-c* pkgs

* chore(NA): correct exclude pattern for each changed pkg

* chore(NA): remove src folder from kbn-y* pkgs

* chore(NA): remove src folder from kbn-e* pkgs

* chore(NA): remove src folder from kbn-f* and kbn-g* pkgs

* chore(NA): remove src folder from kbn-f* and kbn-g* pkgs

* chore(NA): remove src folder from kbn-h** pkgs

* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'

* Revert "chore(NA): remove src folder from kbn-h** pkgs"

This reverts commit dcdf72bcc1.

* fix(NA): grammar location

* test(NA): fix tests for kbn/config-schema

* test(NA): fix tests for kbn/config-schema

* chore(NA): multiple errors fixed

* chore(NA): remove kuery grammar fix

* fix(NA): @kbn/ace imports

* fix(NA): grammar location

* fix(NA): add missing files to tsconfigs

* [CI] Auto-commit changed files from 'node scripts/build_plugin_list_docs'

* chore(NA): complete tsconfigs

* Revert "chore(NA): complete tsconfigs"

This reverts commit f48c616864.

* chore(NA): remove src folder from kbn-core* pkgs

* chore(NA): remove src folder from kbn-u* pkgs

* chore(NA): remove src folder from kbn-ui-shared-deps* pkgs

* chore(NA): fix problems on core pkgs

* chore(NA): fix problems on core pkgs

* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'

* fix(NA): shared_built_assets correct location'

* test(NA): update @kbn/optimizer integration snapshots

* chore(NA): remove src folder from kbn-t* p1 pkgs

* chore(NA): remove src folder from kbn-t* p2 pkgs

* chore(NA): update rootDir on utility types pkg

* chore(NA): include missing files on @kbn/test

* chore(NA): include missing files on @kbn/test

* fix(NA): new jest preset paths on @kbn/test

* chore(NA): fix eslint

* fix(NA): new jest preset paths on @kbn/test

* chore(NA): remove exclusion for mocks folde on @kbn/test

* fix(NA): several imports from target_node

* fix(NA): @kbn/test mocha and jest types clash

* chore(NA): remove src folder from kbn-storybook pkg

* chore(NA): remove src folder from kbn-stdio-dev-helpers pkg

* chore(NA): remove src folder from kbn-std pkg

* chore(NA): remove src folder from kbn-sort-pkg-json pkg

* chore(NA): remove src folder from kbn-some-dev-log and kbn-shared-ux-utility pkgs

* chore(NA): remove src folder from kbn-ux-storybook pkg

* chore(NA): remove src folder from kbn-shared-ux-services pkg

* chore(NA): remove src folder from kbn-shared-ux-components pkg

* chore(NA): remove src folder from kbn-shared-svg pkg

* chore(NA): remove src folder from kbn-server-http-tools pkg

* chore(NA): remove src folder from kbn-securitysolution-* pkgs

* chore(NA): remove src folder from kbn-r-* pkgs

* chore(NA): remove src folder from kbn-p* pkgs

* chore(NA): remove src folder from kbn-o* pkgs

* chore(NA): remove src folder from kbn-m* pkgs

* chore(NA): remove src folder from kbn-j,k,l* pkgs

* chore(NA): remove src folder from kbn-j,k,l* pkgs

* chore(NA): remove src folder from kbn-io-ts-utils* pkgs

* chore(NA): remove src folder from kbn-* pkgs except a few

* chore(NA): update @kbn/generate

* fix(NA): wrong exclusion on kbn-storybook

* chore(NA): remove src folder from kbn-monaco pkg

* chore(NA): remove src folder from kbn-interpreter pkg

* fix(NA): wrong exclusion on kbn-storybook

* chore(NA): update every require for target_*/src

* chore(NA): remover src folder from @kbn/handlebars

* fix(NA): license for @kbn/handlebars

* chore(NA): copy templates as part of the jsts_transpiler macro for @kbn/storybook

* chore(NA): update handlebars

* fix(NA): @kbn/plugin-generator import paths

* fix(NA): bundle sizes

* fix(NA): web bundle for @kbn/i18n-react

* Revert "fix(NA): bundle sizes"

This reverts commit 8aefe84fbc.

* Revert "Revert "fix(NA): bundle sizes""

This reverts commit e9d87d72a4.

* fix(NA): @kbn/docs-utils index.ts path expectation

* chore(NA): merge and solve conflicts with main

* fix(NA): relative import to index

* chore(NA): merge and solve conflicts with main

* [CI] Auto-commit changed files from 'node scripts/precommit_hook.js --ref HEAD~1..HEAD --fix'

* chore(NA): apply eslint fix

* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2022-08-30 15:57:35 +01:00
Tiago Costa
df8b62aa98
chore(NA): allow packages to not have an obligation for a src folder on packages (#138038)
* refact(NA): apply root_input_dir=src to each already created pkg

* refact(NA): update package generator

* fix(NA): correctly use rootDir

* fix(NA): use root input dir on latest introduced pkgs for jsts_transpiler macro

* chore(NA): merge with main

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2022-08-08 15:54:05 +01:00
Jonathan Budzenski
15f194143b
Replace npm zlib with native zlib (#136139)
This is built-in to node, we no longer need to use this package.
2022-07-11 14:56:25 -05:00
Spencer
a3b2757e4e
[type-summarizer] reimplement for broader support (#135163)
* [type-summarizer] reimplement for broader support

* Enable sourceMaps in all packages

* include naming collision in summarizePackage test

* fix readmes

* remove unnecessary transient dependency

* remove code that was commented out

* remove outdated todo comment

* ensure errors triggered by untyped-exports are ligible

* remove unused import

* break out snippet generation from AstIndexer

* refactor several massive files into smaller pieces and add more inline docs

* fix typos

* update jest snapshots

* add sections to readme that points people to the useful parts of the source code along with a high-level overview of how the type-summarizer works

* remove --dump flag, it doesn't work

* use decName instead of calling names.get a second time

* include `export` as invalid name
2022-07-06 13:48:45 -05:00
Kevin Lacabane
4c4f0f5d79
esArchiver datastream support (#132853)
* aliases fallback

* nasty datastream support implementation

* datastreams stats method

* update filter stream

* datastream support for unload action

* create-index datastream support

* index records data stream support

* doc records data streams support

* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'

* lint

* pull composable templates

* set data_stream as a separate property on documents

* force create bulk operation when datastream record

* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'

* lint

* getIndexTemplate tests

* [CI] Auto-commit changed files from 'node scripts/precommit_hook.js --ref HEAD~1..HEAD --fix'

* share cache across transform executions

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2022-06-02 19:38:38 +02:00
Spencer
f3d69b8197
[@kbn/dev-utils] break out more pieces (#132292)
* [@kbn/dev-utils] break out more pieces

* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2022-05-17 11:19:20 -05:00
Spencer
4aa3f8a8ad
[ftr] improve --config error handling and remove default (#131832) 2022-05-09 12:38:38 -07:00
Spencer
542b381fa5
[ftr] automatically determine config run order (#130983)
* [ftr] automatically determine config run order

* split lens config into two groups

* support ftr configs always running against CI

* Split detection_engine_api_integration rule exception list tests

* Add configs from previous commit

* [ftr] remove testMetadata and maintain a unique lifecycle instance per run

* Revert "[ftr] remove testMetadata and maintain a unique lifecycle instance per run"

This reverts commit d2b4fdb824.

* Split alerting_api_integration/security_and_spaces tests

* Add groups to yaml

* Revert "Revert "[ftr] remove testMetadata and maintain a unique lifecycle instance per run""

This reverts commit 56232eea68.

* stop ES more forcefully and fix timeout

* only cleanup lifecycle phases when the cleanup is totally complete

* only use kill when cleaning up an esTestInstance

* fix broken import

* fix runOptions.alwaysUseSource implementation

* fix config access

* fix x-pack/ccs config

* fix ml import file paths

* update kibana build id

* revert array.concat() change

* fix baseConfig usage

* fix pie chart data

* split up maps tests

* pull in all of group5 so that es archives are loaded correctly

* add to ftr configs.yml

* fix pie chart data without breaking legacy version

* fix more pie_chart stuff in new vis lib

* restore normal PR tasks

* bump kibana-buildkite-library

* remove ciGroup validation

* remove the script which is no longer called from checks.sh

* [CI] Auto-commit changed files from 'yarn kbn run build -i @kbn/pm'

* adapt flaky test runner scripts to handle ftrConfig paths

* fix types in alerting_api_integration

* improve flaky config parsing and use non-local var name for passing explicit configs to ftr_configs.sh

* Split xpack dashboard tests

* Add configs

* [flaky] remove key from ftr-config steps

* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'

* restore cypress builds

* remove ciGroups from FTR config files

* fixup some docs

* add temporary script to hunt for FTR config files

* use config.base.js naming for clarity

* use script to power ftr_configs.yml

* remove usage of removed x-pack/scripts/functional_tests

* fix test names in dashboard snapshots

* bump kibana-buildkite-library

* Try retrying only failed configs

* be a little quieter about trying to get testStats from configs with testRunners defined

* Remove test code

* bump kibana-buildkite-library

* update es_snapshot and on_merge jobs too

* track duration and exit code for each config and print it at the end of the script

* store results in order, rather than by key, in case there are duplicates in $config

* bash is hard

* fix env source and use +e rather than disabling e for whole file

* bash sucks

* print config summary in jest jobs too

* define results in jest_parallel.sh

* simplify config summary print, format times a little better

* fix reference to unbound time variable, use better variable name

* skip the newline between each result

* finish with the nitpicking

* sync changes with ftr_configs.sh

* refuse to execute config files which aren't listed in the .buildkite/ftr_configs.yml

* fix config.edge.js base config import paths

* fix some readmes

* resolve paths from ftr_configs manifest

* fix readConfigFile tests

* just allow __fixtures__ configs

* list a few more cypress config files

* install the main branch of kibana-buildkite-library

* split up lens group1

* move ml data_visualizer tests to their own config

* fix import paths

* fix more imports

* install specific commit of buildkite-pipeline-library

* sort configs in ftr_configs.yml

* bump kibana-buildkite-library

* remove temporary script

* fix env var for limiting config types

* Update docs/developer/contributing/development-functional-tests.asciidoc

Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>

* produce a JUnit report for saved objects field count

* apply standard concurrency limits from flaky test runner

* support customizing FTR concurrency via the env

Co-authored-by: Brian Seeders <brian.seeders@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
2022-05-04 17:05:58 -05:00
Spencer
2a78f350e1
break out parts of @kbn/dev-utils (#130509)
* break out parts of @kbn/dev-utils

* autofix imports and kbn/pm dist

* update readme for @kbn/stdio-dev-helpers

* finish renames
2022-04-19 12:24:58 -05:00
Spencer
614139b8e5
[bazel] avoid a little boilerplate in packages (#126309)
* [bazel] avoid a little boilerplate for @types packages

* [bazel/ts] stop building sourcemaps since they're ignored
2022-02-24 12:43:12 -06:00
Tiago Costa
a5b4b822ce
chore(NA): splits types from code on @kbn/test (#124469)
* chore(NA): splits types from code on @kbn/test

* chore(NA): create new @kbn/test-jest-helpers

* chore(NA): move wrong files into @kbn/test

* chore(NA): remove @kbn/test/jest references

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2022-02-04 03:27:39 +00:00
Spencer
12e63dd469
[ftr] support filtering tests by es version (#123289)
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2022-01-19 15:46:32 -07:00
Mikhail Shustov
38feafad72
Update ES client to canary.37 (#119791)
* upgrade es client to canary 37

* fix error in core

* mute error in test/

* mute incompatible errors

* unskip request_entity_too_large_exception test

* commit autofix

* unskip batch_size_bytes_exceeds_es_content_length test

* fix errors in Core code

* fix or mute errors in data plugin

* fix data_view_management

* fix error index_management

* fix x-pack/test errors

* fix watcher

* fix event_log

* fix data_enhanced

* fix uptime

* fix triggers_actions_ui

* fix transform

* fix reporting

* fix rule_registry

* fix timeline

* fix task_manager

* fix security_solution

* fix rule_registry

* fix fleet

* fix index_management

* fix lens

* fix maps

* fix ml

* fix observability

* bump to canary 8.1-2

* fix error in packages

* fix errors in core

* fix errors in data

* fix errors in discover

* fix some discover tests

* fix errors in telemetry

* fix alerting

* fix errors in apm

* fix errors in event_log

* fix errors in fleet

* fix errors in infra

* fix errors in lists

* mute errors in maps

* fix errors in ml

* fix errors in osquery

* fix errors in security_solution

* fix errors in stack_alerts

* fix errors in task_manager

* fix errors in timelines

* fix errors in transform

* fix errors in watcher

* fix errors in tests

* update docs

* adjust fleet code

* fix problem in fleet

* remove outdated _type from data plugin tests

* fix packages tests

* update type

* fix tests part 2

* Adds product header back to elasticsearch

* Updates API docs

* Fix Discover tests

* update alerting typings

* Fix type errors

* Import Sort type

* Update x-pack/plugins/security_solution/common/endpoint/data_loaders/index_fleet_server.ts

Co-authored-by: Steph Milovic <stephanie.milovic@elastic.co>

* Remove double negation

* Fix snapshot conflict issues after merge from "main"

Co-authored-by: Christiane Heiligers <christiane.heiligers@elastic.co>
Co-authored-by: Matthias Wilhelm <matthias.wilhelm@elastic.co>
Co-authored-by: Dario Gieselaar <dario.gieselaar@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Alejandro Fernández Haro <afharo@gmail.com>
Co-authored-by: Steph Milovic <stephanie.milovic@elastic.co>
Co-authored-by: Alejandro Fernández Haro <alejandro.haro@elastic.co>
2021-12-24 14:43:53 +01:00
Tiago Costa
64468b7a3e
chore(NA): splits types from code on @kbn/utils (#121924) 2021-12-23 03:18:10 +00:00
Tyler Smalley
744849f491
Support system indices being hidden (#120985)
Co-authored-by: Josh Dover <doverfake@elastic.co>
2021-12-09 14:23:27 -07:00
Tiago Costa
98f41aaf57
chore(NA): splits types from code on @kbn/es-archiver (#120657) 2021-12-08 02:33:32 +00:00
Tiago Costa
de5dd787b1
chore(NA): splits types from code on @kbn/dev-utils (#120531)
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-12-07 20:12:18 +00:00
Joe Portner
349434ed04
Fix cluster call responses that are missing source fields (#118642)
Change usage from `_source: 'true'` to `_source: true`.
2021-11-16 08:51:22 -05:00
Thomas Watson
788db0dd9d
Remove bluebird dependency (#118097) 2021-11-11 15:17:02 +01:00
Joe Portner
f4b61d01be
Improve es-archiver saving/loading (#118255) 2021-11-10 22:59:38 -05:00
Spencer
4385ac4d83
[eslint] enable type-specific lint rules (#114184)
* [eslint] enable type-specific lint rules

* autofix violations

* duplicate eslint-disable to new export statement

Co-authored-by: spalger <spalger@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-11-03 16:56:17 -06:00
Mikhail Shustov
3c8fa527a7
[ES] Upgrade client to v8.0 (#113950)
* bump to a pre-8.0 version

* export KibanaClient from /lib sub-folder

* workaround the problem of the absence of estypes

* update es client usage in pacakges

* export estypes from another path

* import errors from root

* import errors from root 2

* update transport import

* update import path for /api/types

* update import path for /api/types

* import errors from top export

* use TransportResult instead if ApiResponse

* fix errors in client_config

* fix src/core/server/saved_objects/migrationsv2/actions/integration_tests/actions.test.ts

* use KibanaClient in mock. we dont export the original Client

* fix client mocks

* fix errors on SO

* fix remaining core errors

* update estype import path

* fix errors in data plugin

* fix data_views

* fix es_ui_shared

* fix errors in interactive_setup

* fix errors in ./test folder

* add @elastic/transport to the runtime deps

* fix errors in packages

* fix erros in src/core

* fix errors in test/

* fix an error in actions plugin

* woraround and fix errors in APM plugin

* fix errors in canvas

* fix errors in event_log

* fix errors in fleet

* fix errors in ILM

* fix errors in infra

* fix errors in ingest_pipeline

* fix errors in lens

* fix errors in license_management

* fix errors in licensing

* fix errors in logstash

* fix errors in ml

* fix errors in monitoring

* fix errors in observability

* fix errors in rule_registry

* fix errors in reporting

* fix errors in rule_registry

* fix errors in security

* fix errors in security_solution

* fix errors in snapshot_restore

* fix errors in transform

* fix errors in UA

* fix errors in uptime

* fix errors in x-pack/test

* fix eslint errors

* fix new errors

* use default HTTP Connection. Undici does not support agent config options keepAlive and maxSockets

* create does not accept require_alias option

* update deps

* use transport types exported from ES client package

* fix ErrorCause | string errors

* do not use enum

* fix errors in data plugin

* update x-pack code

* fix transport

* fix apm search request

* do not crash on reporting

* fix kbn-test build

* mute reporting error to start

* fix ftr build

* another attempt

* update import path

* address or mute new errors

* REMOVE me. pin transport version temporarily.

* remove deep imports from transport package

* fix jest crash

* fix product check tests

* remove unnecessary ts-expect-error

* fix a few failed unit tests

* bump to canary 24

* remove unnecessary ts-expect-error

* remove dependency on transport

* fix types in tests

* mute errors in xpack tests

* product check doesn;t  spam in logs anymore

* filterPath --> filter_path

* ignoreUnavailable --> ignore_unavailable

* ignoreUnavailable --> ignore_unavailable

* trackScores --> track_scores

* trackTotalHits --> track_total_hits

* fix es-arcives

* fix data plugin crashes

* fix watcher test utils

* rollback unnecessary changes

* fix another problem in es-archiver

* fix scroll. for whatever reason scroll fails when request scroll_id in body

* add meta: true in kbn-securitysolution-es-utils

* bump client to canary 25

* fix errors in accordance with the es client spec

* update securityscolution-es-utils

* unify scroll api in reporting and fix tests

* fix unit tests in watcher

* refactor APM to abort request with AbortController API

* fix missing es client calls in tests

* fix missing meta in detection engine FTR tests

* fix another bunch of errors in js tests

* fix wrong coercion

* remove test-grep pattern

* fix apm unit test

* rename terminateAfter to terminate_after in infra plugin

* rename terminateAfter to terminate_after in uptime plugin

* rename terminateAfter to terminate_after in apm plugin

* fix security roles FTR tests

* fix reference

* fix post_privilidges test

* fix post_privilidges

* bump client to 26

* add meta for index_management test helpers

* remove ts-expect-error caused by bad type in reason

* bump client to 27

* REMOVE me. workaround until fixed in the es client

* fix incorrect type casting

* swtich from camelCase params

* use `HttpConnection` for FTR-related clients

* bump client to 29

* Revert "REMOVE me. workaround until fixed in the es client"

This reverts commit c038850c09.

* fix new util

* revert repository changes

* do not crash if cannot store event_loop data

* fix new estypes imports

* fix more types

* fix security test types and add ts-ignore for custom ES client

* fix more estypes imports

* yet more ts violations

* line by line fixing is hard

* adapt `evaluateAlert` from infra as it's also used from FTR tests

* use convertToKibanaClient in FTR test instead of meta:true in plugin code

* migrate from deprecated API in fleet

* fix intergration tests

* fix fleet tests

* fix another fleet test

* fix more tests

* let's call it a day

* Removes custom header check on 404 responses, includes es client ProductNotSupportedError in EsUnavailableError conditional (#116029)

* Removes custom header check on 404 responses, includes es client ProductNotSupportedError in EsUnavailableError conditional

* Updates proxy response integration test

* disable APM until compatible with client v8

* skip async_search FTR test

* use kbnClient in integration tests

* bump version to 29

* bump to 30

* have configureClient return a KibanaClient instead of Client, remove resolved violations.

* bump to 31

* bump to 31

* Revert "bump to 31"

This reverts commit 5ac713e640.

* trigger stop to unusubscribe

* update generated docs

* remove obsolete test

* put "as" back

* cleanup

* skip test

* remove new type errors in apm package

* remove ErrorCause casting

* update a comment

* bump version to 32

* remove unnecessary ts-expect-error in apm code

* update comments

* update to client v33

* remove outdated type definition

* bump to 34 without params mutation

* unskip the test that should not fail anymore

* remove unnecessary ts-expect-error comments

* update to v35. body can be string

* move `sort` to body and use body friendly syntax

* fix a failing test. maps register the same SO that has been already registered by home

Co-authored-by: pgayvallet <pierre.gayvallet@gmail.com>
Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
2021-10-26 14:08:22 +02:00
Tyler Smalley
d20a696e95
[ci-stats] Collects additional timings about bootstrap (#112919)
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
2021-09-28 20:56:49 -04:00
Tiago Costa
7c4e4f507b
chore(NA): replace babel config files by inline presets by default in the jsts_transpiler rule (#110620)
* chore(NA): replace babel config files by inline presets by default in the js_ts_transpiler rule

* chore(NA): update @kbn/ace build to exclude worker file

* chore(NA): remove config file support

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-09-03 21:17:46 +01:00
Tiago Costa
560bd0b57b
chore(NA): moving @kbn/es-archiver to babel transpiler (#108370)
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-08-13 18:38:56 +01:00
Spencer
f236286b62
[kbn/es-archiver] fix flaky test (#108143)
Co-authored-by: spalger <spalger@users.noreply.github.com>
2021-08-11 11:51:48 -07:00
spalger
441fb796e2 skip flaky test (#108043) 2021-08-10 23:49:47 -07:00
Spencer
c0395c9ef6
[build_ts_refs] improve caches, allow building a subset of projects (#107981)
* [build_ts_refs] improve caches, allow building a subset of projects

* cleanup project def script and update refs in type check script

* rename browser_bazel config to avoid kebab-case

* remove execInProjects() helper

* list references for tsconfig.types.json for api-extractor workload

* disable composite features of tsconfig.types.json for api-extractor

* set declaration: true to avoid weird debug error

* fix jest tests

Co-authored-by: spalger <spalger@users.noreply.github.com>
2021-08-10 22:12:45 -07:00
Tiago Costa
5f35d23416
chore(NA): moving @kbn/es-archiver into bazel (#103770) 2021-06-29 23:33:02 -04:00
Jonathan Budzenski
bed5b6d8f2
[packages] Migrate @kbn/test to Bazel (#103122)
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-06-29 21:16:00 -04:00
Tyler Smalley
00c180787f
[FTR] Stabilize SSLP functional tests (#102553)
* Removes spaces check, since spaces should always be available
* Disables Monitoring, SecuritySolutions, and Reporting in SSPL tests
until #102552 is completed

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
2021-06-17 16:47:00 -04:00
Spencer
f466ebf1a3
[esArchiver] drop support for --dir, use repo-relative paths instead (#101345)
Co-authored-by: spalger <spalger@users.noreply.github.com>
2021-06-08 17:37:42 -04:00
Mikhail Shustov
d920682e4e
Update @elastic/elasticsearch to 8.0.0-canary13 (#98266)
* bump @elastic/elasticsearch to canary.7

* address errors in core

* address errors in data plugin

* address errors in Alerting team plugins

* remove outdated messages in Lens

* remove unnecessary comments in ML

* address errors in Observability plugin

* address errors in reporting plugin

* address errors in Rule registry plugin

* fix errors in Security plugins

* fix errors in ES-UI plugin

* remove unnecessary union.

* update core tests

* fix kbn-es-archiver

* update to canary 8

* bump to v9

* use new typings

* fix new errors in core

* fix errors in core typeings

* fix type errors in data plugin

* fix type errors in telemetray plugin

* fix data plugin tests

* fix search examples type error

* fix errors in discover plugin

* fix errors in index_pattern_management

* fix type errors in vis_type_*

* fix errors in typings/elasticsearch

* fix type errors in actions plugin

* fix type errors in alerting and apm plugins

* fix type errors in canvas and cases

* fix errors in event_log

* fix type errors in ILM and ingest_pipelines

* fix errors in lens plugin

* fix errors in lists plugin

* fix errors in logstash

* fix errors in metrics_entities

* fix errors in o11y

* fix errors in watcher

* fix errors in uptime

* fix errors in upgrade_assistant

* fix errors in task_manager

* fix errors in stack_alerts

* fix errors in security_solution

* fix errors in rule_registry

* fix errors in snapshot_restore

* fix remaining errors

* fix search intergration tests

* adjust assetion

* bump version to canary.10

* adapt code to new naming schema

* use mapping types provided by the client library

* Revert "adjust assetion"

This reverts commit 19b8fe0464.

* fix so intergration tests

* fix http integration tests

* bump version to canary 11

* fix login test

* fix http integration test

* fix apm test

* update docs

* fixing some ml types

* fix new errors in data plugin

* fix new errors in alerting plugin

* fix new errors in lists plugin

* fix new errors in reporting

* fix or mute errors in rule_registry plugin

* more ML type fixes

* bump to canary 12

* fix errors after merge conflict

* additional ML fixes

* bump to canary 13

* fix errors in apm plugin

* fix errors in fleet plugin

* fix errors in infra plugin

* fix errors in monitoring plugin

* fix errors in osquery plugin

* fix errors in security solution plugins

* fix errors in transform plugin

* Update type imports for ES

* fix errors in x-pack plugins

* fix errors in tests

* update docs

* fix errors in x-pack/test

* update error description

* fix errors after master merge

* update comment in infra plugin

* fix new errors on xpack tests/

Co-authored-by: James Gowdy <jgowdy@elastic.co>
Co-authored-by: Dario Gieselaar <dario.gieselaar@elastic.co>
2021-06-08 15:06:06 +02:00
spalger
66867bbede Revert a terrible mistake
Revert "save"

This reverts commit 4272bfb972.
2021-05-27 14:27:42 -07:00
spalger
4272bfb972 save 2021-05-27 07:33:11 -07:00
Tyler Smalley
537be25754
[FTR][CI] Use default distribution for all tests (#94968)
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
2021-05-12 15:24:25 -04:00