Ensure that a plugin containing a nested `node_modules` folder correctly
identifies imports from that folder as regular node modules instead of relative imports.
Kibana should currently only have a single root `node_modules` folder, so this should
not be a real issue. However, if for some reason a stray `node_modules` folder is
created, this commit will make sure it's not introducing hard to debug ESLint errors.
Includes a few compiled cypress typescript files generated during `node
scripts/type_check`. We're not using a global glob due to some plugins
using a javascript base.
This adds an overview of the number of documents per document type. It's
possible to filter the data via search bar and time range.
<img width="1478" alt="image"
src="32c1c79c-c76a-4b84-ab16-c1b1faccba32">
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Pitch: https://github.com/elastic/apm-dev/issues/872 (_internal_)
Introduces the Diagnostics tool under a hidden page:
`/app/apm/diagnostics`. The Diagnostics tool can help find problems with
missing index templates, data streams with incorrect index templates and
indices with incorrect mappings.
It will be released in 8.9 but customers can take advantage of it as
soon as this PR is merged. They can export a diagnostics bundle from
their system by running this script:
```
node x-pack/plugins/apm/scripts/diagnostics_bundle \
--kbHost https://foo.kb.europe-west2.gcp.elastic-cloud.com:9243 \
--esHost https://foo.es.europe-west2.gcp.elastic-cloud.com:9243 \
--username elastic
--password changeme
```
---------
Co-authored-by: Yngrid Coello <yngrdyn@gmail.com>
## Summary
This PR adds boilerplate code and a few initial end-to-end tests to
serverless plugins.
Note that the tests defined in this PR are not part of any CI run yet,
this will be done in a follow-up after this PR is merged.
### Details
The serverless test structure corresponds to what we have in
`x-pack/test` with API tests in `api_integration` and UI tests in
`functional`, each with their set of helper methods and sub-directories
for
- `common` functionality shared across serverless projects (core, shared
UX, ...)
- `observability` project specific functionality
- `search` project specific functionality
- `security` project specific functionality
The `shared` directory contains fixtures, services, ... that are shared
across `api_integration` abd `functional` tests.
```
x-pack/test_serverless/
├─ api_integration
│ ├─ services
│ ├─ test_suites
│ │ ├─ common
│ │ ├─ observability
│ │ ├─ search
│ │ ├─ security
├─ functional
│ ├─ page_objects
│ ├─ services
│ ├─ test_suites
│ │ ├─ common
│ │ ├─ observability
│ │ ├─ search
│ │ ├─ security
├─ shared
│ ├─ services
│ ├─ types
```
See also `x-pack/test_serverless/README.md`
### Run tests
Similar to how functional tests are run in `x-pack/test`, you can point
the functional tests server and test runner to config files in this
`x-pack/test_serverless` directory, e.g. from the `x-pack` directory
run:
```
node scripts/functional_tests_server.js --config test_serverless/api_integration/test_suites/common/config.ts
```
and
```
node scripts/functional_test_runner.js --config test_serverless/api_integration/test_suites/common/config.ts
```
### Additional changes
- The stateful `common_page` page object used the existence of the
global nav to determine `isChromeVisible` and `isChromeHidden`, which is
not working when the global nav is disabled. To solve this, a
`data-test-subj` that indicates the chrome visible state is added to the
Kibana app wrapper and is used for the checks.
- Add a few `data-test-subj` entries to the Observability overview page.
- Add optional `dataTestSubj` to the `Navigation` component and use that
for the serverless search nav.
- Add optional `titleDataTestSubj` to the `SolutionNav` component and
use it for the serverless security nav.
- Add a data-test-subj entry to the Search overview page.
Fixes https://github.com/elastic/kibana/issues/149344
This PR migrates all plugins to packages automatically. It does this
using `node scripts/lint_packages` to automatically migrate
`kibana.json` files to `kibana.jsonc` files. By doing this automatically
we can simplify many build and testing procedures to only support
packages, and not both "packages" and "synthetic packages" (basically
pointers to plugins).
The majority of changes are in operations related code, so we'll be
having operations review this before marking it ready for review. The
vast majority of the code owners are simply pinged because we deleted
all `kibana.json` files and replaced them with `kibana.jsonc` files, so
we plan on leaving the PR ready-for-review for about 24 hours before
merging (after feature freeze), assuming we don't have any blockers
(especially from @elastic/kibana-core since there are a few core
specific changes, though the majority were handled in #149370).
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
We are currently expecting serverless environments to be broken up into
several different envs, this config structure allows us to customize the
config based on that environment without major modifications to the
config loading system.
This adds the `--serverless` CLI arg (only available in dev mode, you
can also use `yarn start-serverless`), which will load the new
`config/kibana.serverless.yml` file. For now, this file is not included
in the build artifact, though we might include a `--serverless` flag
which replaces the `kibana.yml` file with `kibana.serverless.yml`.
@jbudz will follow up after this PR with the build related changes to
get this working with PR cloud deploys, which will be enough changes for
us to start iterating on UI specific changes based on running in a
serverless environment.
Additionally, support for the undocumented `KBN_CONFIG_PATHS` env var is
added, which should contain a comma-separated list of paths to kibana
config files. These files are loaded, in the specified order, before any
of the config files listed in the CLI but after the kibana.yml config
file.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
The location of plugins was previously somewhat irrelevant, but as we
move into packages it's more important that we can find all plugins in
the repository, and we would like to be able to do that without needing
to maintain a manifest somewhere to accomplish this. In order to make
this possible we plan to find any plugin/package by spotting all
kibana.json files which are not "fixtures". This allows plugin-like code
(but not actual plugin code) to exist for testing purposes, but it must
be within some form of "fixtures" directory, and any plugin that isn't
in a fixtures directory will be automatically pulled into the system
(though test plugins, examples, etc. will still only be loaded when the
plugin's path is passed via `--plugin-path`, the system will know about
them and use that knowledge for other things).
Since this is just a rename Operations will review and merge by EOD Jan
12th unless someone has a blocking concern.
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This PR implements a linter like the TS Project linter, except for
packages in the repo. It does this by extracting the reusable bits from
the TS Project linter and reusing them for the project linter. The only
rule that exists for packages right now is that the "name" in the
package.json file matches the "id" in Kibana.jsonc. The goal is to use a
rule to migrate kibana.json files on the future.
Additionally, a new rule for validating the indentation of tsconfig.json
files was added.
Validating and fixing violations is what has triggered review by so many
teams, but we plan to treat those review requests as notifications of
the changes and not as blockers for merging.
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Dearest Reviewers 👋
I've been working on this branch with @mistic and @tylersmalley and
we're really confident in these changes. Additionally, this changes code
in nearly every package in the repo so we don't plan to wait for reviews
to get in before merging this. If you'd like to have a concern
addressed, please feel free to leave a review, but assuming that nobody
raises a blocker in the next 24 hours we plan to merge this EOD pacific
tomorrow, 12/22.
We'll be paying close attention to any issues this causes after merging
and work on getting those fixed ASAP. 🚀
---
The operations team is not confident that we'll have the time to achieve
what we originally set out to accomplish by moving to Bazel with the
time and resources we have available. We have also bought ourselves some
headroom with improvements to babel-register, optimizer caching, and
typescript project structure.
In order to make sure we deliver packages as quickly as possible (many
teams really want them), with a usable and familiar developer
experience, this PR removes Bazel for building packages in favor of
using the same JIT transpilation we use for plugins.
Additionally, packages now use `kbn_references` (again, just copying the
dx from plugins to packages).
Because of the complex relationships between packages/plugins and in
order to prepare ourselves for automatic dependency detection tools we
plan to use in the future, this PR also introduces a "TS Project Linter"
which will validate that every tsconfig.json file meets a few
requirements:
1. the chain of base config files extended by each config includes
`tsconfig.base.json` and not `tsconfig.json`
1. the `include` config is used, and not `files`
2. the `exclude` config includes `target/**/*`
3. the `outDir` compiler option is specified as `target/types`
1. none of these compiler options are specified: `declaration`,
`declarationMap`, `emitDeclarationOnly`, `skipLibCheck`, `target`,
`paths`
4. all references to other packages/plugins use their pkg id, ie:
```js
// valid
{
"kbn_references": ["@kbn/core"]
}
// not valid
{
"kbn_references": [{ "path": "../../../src/core/tsconfig.json" }]
}
```
5. only packages/plugins which are imported somewhere in the ts code are
listed in `kbn_references`
This linter is not only validating all of the tsconfig.json files, but
it also will fix these config files to deal with just about any
violation that can be produced. Just run `node scripts/ts_project_linter
--fix` locally to apply these fixes, or let CI take care of
automatically fixing things and pushing the changes to your PR.
> **Example:** [`64e93e5`
(#146212)](64e93e5806)
When I merged main into my PR it included a change which removed the
`@kbn/core-injected-metadata-browser` package. After resolving the
conflicts I missed a few tsconfig files which included references to the
now removed package. The TS Project Linter identified that these
references were removed from the code and pushed a change to the PR to
remove them from the tsconfig.json files.
## No bazel? Does that mean no packages??
Nope! We're still doing packages but we're pretty sure now that we won't
be using Bazel to accomplish the 'distributed caching' and 'change-based
tasks' portions of the packages project.
This PR actually makes packages much easier to work with and will be
followed up with the bundling benefits described by the original
packages RFC. Then we'll work on documentation and advocacy for using
packages for any and all new code.
We're pretty confident that implementing distributed caching and
change-based tasks will be necessary in the future, but because of
recent improvements in the repo we think we can live without them for
**at least** a year.
## Wait, there are still BUILD.bazel files in the repo
Yes, there are still three webpack bundles which are built by Bazel: the
`@kbn/ui-shared-deps-npm` DLL, `@kbn/ui-shared-deps-src` externals, and
the `@kbn/monaco` workers. These three webpack bundles are still created
during bootstrap and remotely cached using bazel. The next phase of this
project is to figure out how to get the package bundling features
described in the RFC with the current optimizer, and we expect these
bundles to go away then. Until then any package that is used in those
three bundles still needs to have a BUILD.bazel file so that they can be
referenced by the remaining webpack builds.
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* Removed file-type from package.json deps.
Updated jimp to 0.16.1.
Updated light & dark dashboard baseline screenshots to resolve failure due to resize degradation.
* Replaced jimp with sharp and pixelmatch. Eliminates several deps, including file-type.
Eliminates quality degradation from image resizing.
* Fixed copy paste error, which lead to pass positives and errors.
* Integrated updateBaselines flag into dashboards report testing.
Fixed sharp resize to fit and not truncate the original image.
* Removed accidental '.only'
* Reverted area chart baseline.
* Removed explicit declaration of sharp from package.json.
* Adjusted dashboard screenshot size in attempt to eliminate layout scaling issues in CI testing.
* Reverted dashboard screenshot size. Lowered tolerance in dashboard PNG reporting tests to force failure and collect CI screenshots.
* Changed session save location for dashboard report images for better CI reporting.
* Fixed small dashboard baseline screenshot with correct aspect ratio.
* Updated 2 basline screenshots from CI. Lowered PNG match tolerance for dashboard screenshots to 1% for investigation of remaining screens.
* Update of dashboard controls light mode baseline.
Change of forced container width size in large dashboard layout test.
* Updated dashboard controls dark mode baseline screenshot.
Adjusted forced container width in large dashboard layout test for troubleshooting.
* Lowering tolerance of PNG compare tests from 9% to 1% for troubleshooting.
Replaced references to dedundant image service with existing, more adopted reporting service.
Ideally, the PNG functions of the reporting service would be merged with the screenshot service as an image service.
* Reverting tolerance of PNG compares to 0.09. Will create GH issues for specific teams to investigate.
* Implemented PngService to provide an accessible single reference to PNG compare testing.
Removed redundant implementations and references.
* Updated service reference and call to png compare function left out of last commit.
* Updated another missed PNG reference.
* Removed superfluous parameter.
* Update after bootstrap
* Updated area chart baseline to match new render with panel filters.
* Reverted limits in dashboard snapshot tests to original values.
* Merge and rebuild.
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [ts] enable sourcemaps in summarized types of @kbn/crypto
* update snapshots
* remove unnecessary exports of @kbn/type-summarizer package
* remove tsc from the build process
* use `@kbn/type-summarizer` to summarize its own types
* add tests for interface and function
* switch to export type where necessary
* ignore __tmp__ in global jest preset
* ignore __tmp__ globally
* remove `@kbn/crypto` types path
* Support installing bundled packages during Fleet setup
* Fix failing API integration test
* Sort packages in test
* Move get bundled packages + add tests
* Fix mock installPackage type
* Don't attempt to install already-installed bundled packageS
* Use unique installation status for bundled packages
* Fix failing tests
* Fix type error
* Fix logic for latest value
* Remove bundled packages from test since they no longer come back from list endpoint
* Change assertion for empty preconfiguration update request
* Add all bundled packages + refactor install to use Promise.allSettled
* Fix name comparison again
* Rework approach to use preferred source at install-time instead
* Add helper script + update bundled packages + allow updates for upload path
* Fix failing test
* Address code review + add additional tests for update cases
* Fix type error
* Remove configOverride concept for now
* Fix another type error 🙃
* Add build step for bundling fleet packages
* Fix unused import in build step
* Keep bundled_packages directory in place to prevent setup failures
* Remove gitkeep + make bundled package directory lookup fault tolerant
* Update gitignore
* Add doc comment + use build.resolvePath
* Add --use-snapshot-registry CLI flag
* Update snapshots
* Rename arg to use-snapshot-epr
* Don't fail the build on failure to bundled packages
* Update all checksums + don't attempt download multiple times
* Skip checksum process for bundled packages for now
* mkdirp destination directory to fix tests
* Revert build_distributables file
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* added cypress setup for fleet
* updated readme
* fixed types
* fix unused
* starting up fleet server in cypress
* added more fleet tests
* added package update test
* updated readme
* trying jenkins run
* trying jenkins run
* trying jenkins run
* fix types, cypress config
* example of mocked requests
* added getBySel
* fix test after refactor
* removed duplication
* fix tests
* added to buildkite
* updated tests
* starting agent with docker
* trying to fix ip address
* ifconfig
* ip address
* ip address
* ip address
* type fix
* remove extra logging
* fixed test
* fix buildkite
* cleanup
* cleanup
* using docker for fleet_server
* fix
* fix
* trying to fix
* update config
* test
* test
* test
* test
* revert changes
* test
* test
* static ip on linux
* docker version
* try again
* fixed review comments
* fixed types
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Patryk Kopycinski <contact@patrykkopycinski.com>
* [build_ts_refs] improve caches, allow building a subset of projects
* cleanup project def script and update refs in type check script
* rename browser_bazel config to avoid kebab-case
* remove execInProjects() helper
* list references for tsconfig.types.json for api-extractor workload
* disable composite features of tsconfig.types.json for api-extractor
* set declaration: true to avoid weird debug error
* fix jest tests
Co-authored-by: spalger <spalger@users.noreply.github.com>
* update api_docs from gitignore
* api doc files
* Update api_welcome.mdx
* Pretty print json
* Add new lines and extra frontmatter to warn github viewers
* update api_docs with new lines and frontmatter
* Remove all apis tagged @internal and add tests
* Update docs with all internal APIs removed
* fix typescript error in test fixtures
* Update core.json
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
This package has long since been deprecated and no longer maintained.
The only reason it still exists is that not all plugins have migrated
away from using the kui classes to EUI. This removes all src and build
files as a step forward.
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
This PR adds screenshot comparison tests for the nodes in the graph on the test plugin.
Run the tests using this command:
`yarn test:ftr --config x-pack/test/plugin_functional/config.ts --grep Resolver`
* chore(NA): update gitignore to include first changes from moving into a single package.json
* chore(NA): update gitignore
* chore(NA): move all the dependencies into the single package.json and apply changes to bootstrap
* chore(NA): fix types problems after the single package json
* chore(NA): include code to find the dependencies used across the code
* chore(NA): introduce pure lockfile for install dependencies on build
* chore(NA): update clean task to not delete anything from xpack node_modules
* chore(NA): update gitignore to remove development temporary rules
* chore(NA): update notice file
* chore(NA): update jest snapshots
* chore(NA): fix whitelisted licenses to include a new specify form of an already included one
* chore(NA): remove check lockfile symlinks from child projects
* chore(NA): fix eslint and add missing declared deps on single pkg json
* chore(NA): correctly update notice
* chore(NA): fix failing jest test for storyshots.test.tsx
* chore(NA): fix cypress multi reporter path
* chore(NA): fix Project tests check
* chore(NA): fix problem with logic to detect used dependes on oss build
* chore(NA): include correct x-pack plugins dep discovery
* chore(NA): discover entries under dynamic requires on vis_type_timelion
* chore(NA): remove canvas
* test(NA): fix jest unit tests
* chore(NA): remove double react declaration from storyshot test file
* chore(NA): try removing isOSS check
* chore(NA): support for plugin development
* chore(NA): update logic to fix unit tests and typechecking
* chore(NA): support to run npm scripts in child kbn projects across all envs
* chore(NA): support github checks reporter on x-pack and remove cpy types as the package correctly provides them
* chore(NA): update cpy version
* chore(NA): include last kbn pm changes
* chore(NA): update style on build_production_projects.ts
* chore(NA): remove any cast fom telemetry opt in stats
* chore(NA): remove del and re-use rm -rf again
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* move test_helpers to the core
* create base tsconfig
* all tsconfigs use the base one
* use test_helpers exposed from the src/core
* move getFieldFormatsRegistry to data plugin
* add test_utils project
* compile types after checkout
* add a stub for platform tsconfig.json
* fix broken import
* fix broken path to the base config
* set tsBuildInfoFile for project without outDir
* do not commit tsbuildinfo file
* do not check output d.ts files
* fix type error
* use separate config to build types
* rollback changes to include paths
* mute import zone error
* rename files to avoid references to tsd
* do not use tsd for type tests
* include all ts files in project
* run buildRefs before type check to ensure the latest version
* store tsbuildinfo locally
* update paths to base config
* comment out core/tsconfig.json
* remove ui path
* fix wrong tsbuildinfo path
* chore(NA): load NODE_OPTIONS from options files across environments
* chore(NA): move node.ci.options to config folder
* docs(NA): update docs to explain how to set node options from the cfg fil
* chore(NA): removed test npm scripts
* fix(NA): typo on setup script for CI
* chore(NA): add debug info
* chore(NA): export options on CI
* chore(NA): remove debug info
* chore(NA): support for configurable config folder using env var
* chore(NA): add node.options file into docker img
* fix(NA): use calculated config dir on node options for ci
* chore(NA): node bin scripts bootstrap and node_with_options implementation for bash
* chore(NA): complete node_with_options scripts with bat version
* chore(NA): add bin/node dev script and remove cli for run_with_node_options
* chore(NA): increase default maxBuffer
* chore(NA): remove run with options script from package.json
* chore(NA): include kbn-node script and underlying usage of it
* chore(NA): remove change on eslint
* chore(NA): correct typo on kbn node script comment
Co-authored-by: Tyler Smalley <tylersmalley@me.com>
* chore(NA): correct typo on kbn node script comment
Co-authored-by: Tyler Smalley <tylersmalley@me.com>
* chore(NA): add line to describe each option should be specified in a separated line
* chore(NA): remove node options from dev and ci env
* chore(NA): remove changes from package.json
* chore(NA): fix docker image build
* chore(NA): change value for example of --max-old-space-size in the node.options file
Co-authored-by: Tyler Smalley <tylersmalley@me.com>
* chore(NA): remove --no-warnings from node.options and force it in the bin scripts
* chore(NA): prevent 'The system cannot find the file' error message
* chore(NA): introduce slash when building path for %DIR%
* chore(NA): read options from file only if it exists
Co-authored-by: Jonathan Budzenski <jbudz@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Tyler Smalley <tylersmalley@me.com>
