Commit graph

244 commits

Author SHA1 Message Date
Tiago Costa
9621d8dd51
[8.16] chore(NA): upgrade to webpack 5 (#191106) (#211152)
# Backport

This will backport the following commits from `main` to `8.16`:
- [chore(NA): upgrade to webpack 5
(#191106)](https://github.com/elastic/kibana/pull/191106)

<!--- Backport version: 9.6.4 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Tiago
Costa","email":"tiago.costa@elastic.co"},"sourceCommit":{"committedDate":"2025-02-14T03:01:36Z","message":"chore(NA):
upgrade to webpack 5 (#191106)\n\n### Summary\r\n\r\n- Closes
#89741\r\n\r\nThis PR contains the resulting work of a massive effort
that ports our\r\non top bundler abstraction (called @kbn/optimizer)
from Webpack v4 into\r\nWebpack v5. It's essential in terms of long term
maintenance since v4\r\nwas not receiving updates any longer but will
also unblock some new\r\nfeatures that could be beneficial for our
future DevEx endeavours.\r\n\r\nNext you can find a small list of all
the accomplished tasks on this\r\njourney.\r\n\r\n### Completed
Tasks\r\n- [x] Upgrade dependencies to match the ones on webpack v5\r\n-
[x] Fix null-loader usages\r\n- [x] Fix raw-loader usages\r\n- [x] Fix
file-loader usages\r\n- [x] Fix url-loader usages\r\n- [x] Fix
`@kbn/optimizer-webpack-helpers` to support webpack v5 \r\n- [x] Adopt
previous webpack v4 polyfill-all strategy
with\r\nnode-polyfill-webpack-plugin\r\n- [x] Fix theme-loader on
@kbn/optimizer\r\n- [x] Migrate configurations and ad-hoc loader options
on all webpack\r\nconfigs from v4 to v5\r\n- [x] Fix @kbn/test jest
resolver for file-loader cases\r\n- [x] Migrate public-path loader on
UiSharedDeps\r\n- [x] Fix all usages of webpack-merge\r\n- [x] Migrate
BundleRemoteModule\r\n- [x] Migrate BundleRemotesPlugin\r\n- [x]
Correctly migrate PopulateBundleCachePlugin\r\n- [x] Correctly migrate
BundleMetricsPlugin\r\n- [x] Check if the profiling plugins still work
(--profile flag)\r\n- [x] Recover if possible the previous webpack v4
cacheGroup chunks\r\nrename to something like
`data.plugin.chunk.0.js`\r\n- [x] Run `/ci` and make sure we get our
first green CI, otherwise work\r\non the errors until we do\r\n- [x]
Profile and solve bottlenecks until we get a cold build\r\nperformance
similar to the one we had on webpack v4
(`node\r\nscripts/build_kibana_platform_plugins --no-cache`).\r\n- [x]
OpenSSL Legacy Warnings: try to remove `--openssl-legacy-provider\r\n`
flags\r\n- [x] Add Webpack to Renovate config\r\n- [x] Explore removing
`NodePolyfillPlugin`\r\n([here](https://www.npmjs.com/package/node-polyfill-webpack-plugin))
and\r\nadd each polyfill needed individually per each webpack config to
check\r\nif we get smaller bundles. If we do it's better to go with the
case by\r\ncase need approach instead of deploying a bunch of polyfills
with\r\nNodePolyfillPlugin. As another alternative, create a custom
smaller\r\nplugin with only the union of all needed polyfills.\r\n- [x]
Evaluate if we want to touch the resolutions on mainFields
and\r\nconditionNames\r\n- [x] Understand why `@import
'src/core/public/mixins'` does not work\r\nanymore (not a problem, we
should use relative paths anyway but we want\r\nto track why it changed
from v4 to v5)\r\n- [x] BUG: Child compilers are having errors hidden
and/or changed from\r\nerror to warning\r\n- [x] Fix license check
for\r\n[Artistic-2.0](https://spdx.org/licenses/Artistic-2.0.html) is
the\r\nlicense
for\r\n[domain-browser](https://github.com/bevry/domain-browser?tab=License-1-ov-file).\r\nThis
package is a dependency
of\r\n[NodePolyfillPlugin](https://www.npmjs.com/package/node-polyfill-webpack-plugin).\r\nArtistic
2.0 license is [classified
as\r\nyellow](https://github.com/elastic/open-source/blob/main/elastic-product-policy.md#yellow-list)\r\nand
should only be used for dev dependencies.\r\n- [x] Make sure
`resourceQuery: { not: /raw/ }` is not necessary on\r\nother webpack
configs like storybook one\r\n- [x] Find what is being wrongly removed
by usedExports optimization;\r\nhint: I believe it is identifying a lot
of exports inside the sync entry\r\nof plugins as unused exports and
removing them. Then `__kbnBootstrap__`\r\ncan't be found\r\n- [x]
Rebalance @kbn/optimizer pickMaxWorkerCount\r\n- [x] Re-open the issue
to fix
sass-warnings\r\n[#190345](https://github.com/elastic/kibana/issues/190345)
or downgrade\r\nsass-loader to v10\r\n- [x] Remove previous esm no parse
rules\r\n- [x] Confirm esm support is working\r\n- [x] Confirm console
override is needed\r\n- [x] Confirm react prod builds on ui shared deps
for distributable\r\n- [x] Remove customization
for\r\n[xyflow](https://github.com/xyflow/xyflow) from webpack
configs\r\n- [x] Clean all the code\r\n- [x] Make sure collected metrics
from stats are still aligned with what\r\nwe were collecting before;
also verify if the modules used for optimizer\r\ncaches etc are well
generated (@kbn/node-libs-browser)\r\n- [x] Fix watch
performance\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by:
Brad White
<brad.white@elastic.co>","sha":"203bc284781cc830049b3f7948c8fcafa8b7d9a1","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["chore","Team:Operations","release_note:skip","Team:Fleet","v9.0.0","backport:all-open","ci:build-webpack-bundle-analyzer","Team:obs-ux-infra_services","v9.1.0"],"title":"chore(NA):
upgrade to webpack
5","number":191106,"url":"https://github.com/elastic/kibana/pull/191106","mergeCommit":{"message":"chore(NA):
upgrade to webpack 5 (#191106)\n\n### Summary\r\n\r\n- Closes
#89741\r\n\r\nThis PR contains the resulting work of a massive effort
that ports our\r\non top bundler abstraction (called @kbn/optimizer)
from Webpack v4 into\r\nWebpack v5. It's essential in terms of long term
maintenance since v4\r\nwas not receiving updates any longer but will
also unblock some new\r\nfeatures that could be beneficial for our
future DevEx endeavours.\r\n\r\nNext you can find a small list of all
the accomplished tasks on this\r\njourney.\r\n\r\n### Completed
Tasks\r\n- [x] Upgrade dependencies to match the ones on webpack v5\r\n-
[x] Fix null-loader usages\r\n- [x] Fix raw-loader usages\r\n- [x] Fix
file-loader usages\r\n- [x] Fix url-loader usages\r\n- [x] Fix
`@kbn/optimizer-webpack-helpers` to support webpack v5 \r\n- [x] Adopt
previous webpack v4 polyfill-all strategy
with\r\nnode-polyfill-webpack-plugin\r\n- [x] Fix theme-loader on
@kbn/optimizer\r\n- [x] Migrate configurations and ad-hoc loader options
on all webpack\r\nconfigs from v4 to v5\r\n- [x] Fix @kbn/test jest
resolver for file-loader cases\r\n- [x] Migrate public-path loader on
UiSharedDeps\r\n- [x] Fix all usages of webpack-merge\r\n- [x] Migrate
BundleRemoteModule\r\n- [x] Migrate BundleRemotesPlugin\r\n- [x]
Correctly migrate PopulateBundleCachePlugin\r\n- [x] Correctly migrate
BundleMetricsPlugin\r\n- [x] Check if the profiling plugins still work
(--profile flag)\r\n- [x] Recover if possible the previous webpack v4
cacheGroup chunks\r\nrename to something like
`data.plugin.chunk.0.js`\r\n- [x] Run `/ci` and make sure we get our
first green CI, otherwise work\r\non the errors until we do\r\n- [x]
Profile and solve bottlenecks until we get a cold build\r\nperformance
similar to the one we had on webpack v4
(`node\r\nscripts/build_kibana_platform_plugins --no-cache`).\r\n- [x]
OpenSSL Legacy Warnings: try to remove `--openssl-legacy-provider\r\n`
flags\r\n- [x] Add Webpack to Renovate config\r\n- [x] Explore removing
`NodePolyfillPlugin`\r\n([here](https://www.npmjs.com/package/node-polyfill-webpack-plugin))
and\r\nadd each polyfill needed individually per each webpack config to
check\r\nif we get smaller bundles. If we do it's better to go with the
case by\r\ncase need approach instead of deploying a bunch of polyfills
with\r\nNodePolyfillPlugin. As another alternative, create a custom
smaller\r\nplugin with only the union of all needed polyfills.\r\n- [x]
Evaluate if we want to touch the resolutions on mainFields
and\r\nconditionNames\r\n- [x] Understand why `@import
'src/core/public/mixins'` does not work\r\nanymore (not a problem, we
should use relative paths anyway but we want\r\nto track why it changed
from v4 to v5)\r\n- [x] BUG: Child compilers are having errors hidden
and/or changed from\r\nerror to warning\r\n- [x] Fix license check
for\r\n[Artistic-2.0](https://spdx.org/licenses/Artistic-2.0.html) is
the\r\nlicense
for\r\n[domain-browser](https://github.com/bevry/domain-browser?tab=License-1-ov-file).\r\nThis
package is a dependency
of\r\n[NodePolyfillPlugin](https://www.npmjs.com/package/node-polyfill-webpack-plugin).\r\nArtistic
2.0 license is [classified
as\r\nyellow](https://github.com/elastic/open-source/blob/main/elastic-product-policy.md#yellow-list)\r\nand
should only be used for dev dependencies.\r\n- [x] Make sure
`resourceQuery: { not: /raw/ }` is not necessary on\r\nother webpack
configs like storybook one\r\n- [x] Find what is being wrongly removed
by usedExports optimization;\r\nhint: I believe it is identifying a lot
of exports inside the sync entry\r\nof plugins as unused exports and
removing them. Then `__kbnBootstrap__`\r\ncan't be found\r\n- [x]
Rebalance @kbn/optimizer pickMaxWorkerCount\r\n- [x] Re-open the issue
to fix
sass-warnings\r\n[#190345](https://github.com/elastic/kibana/issues/190345)
or downgrade\r\nsass-loader to v10\r\n- [x] Remove previous esm no parse
rules\r\n- [x] Confirm esm support is working\r\n- [x] Confirm console
override is needed\r\n- [x] Confirm react prod builds on ui shared deps
for distributable\r\n- [x] Remove customization
for\r\n[xyflow](https://github.com/xyflow/xyflow) from webpack
configs\r\n- [x] Clean all the code\r\n- [x] Make sure collected metrics
from stats are still aligned with what\r\nwe were collecting before;
also verify if the modules used for optimizer\r\ncaches etc are well
generated (@kbn/node-libs-browser)\r\n- [x] Fix watch
performance\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by:
Brad White
<brad.white@elastic.co>","sha":"203bc284781cc830049b3f7948c8fcafa8b7d9a1"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/211145","number":211145,"state":"OPEN"},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/191106","number":191106,"mergeCommit":{"message":"chore(NA):
upgrade to webpack 5 (#191106)\n\n### Summary\r\n\r\n- Closes
#89741\r\n\r\nThis PR contains the resulting work of a massive effort
that ports our\r\non top bundler abstraction (called @kbn/optimizer)
from Webpack v4 into\r\nWebpack v5. It's essential in terms of long term
maintenance since v4\r\nwas not receiving updates any longer but will
also unblock some new\r\nfeatures that could be beneficial for our
future DevEx endeavours.\r\n\r\nNext you can find a small list of all
the accomplished tasks on this\r\njourney.\r\n\r\n### Completed
Tasks\r\n- [x] Upgrade dependencies to match the ones on webpack v5\r\n-
[x] Fix null-loader usages\r\n- [x] Fix raw-loader usages\r\n- [x] Fix
file-loader usages\r\n- [x] Fix url-loader usages\r\n- [x] Fix
`@kbn/optimizer-webpack-helpers` to support webpack v5 \r\n- [x] Adopt
previous webpack v4 polyfill-all strategy
with\r\nnode-polyfill-webpack-plugin\r\n- [x] Fix theme-loader on
@kbn/optimizer\r\n- [x] Migrate configurations and ad-hoc loader options
on all webpack\r\nconfigs from v4 to v5\r\n- [x] Fix @kbn/test jest
resolver for file-loader cases\r\n- [x] Migrate public-path loader on
UiSharedDeps\r\n- [x] Fix all usages of webpack-merge\r\n- [x] Migrate
BundleRemoteModule\r\n- [x] Migrate BundleRemotesPlugin\r\n- [x]
Correctly migrate PopulateBundleCachePlugin\r\n- [x] Correctly migrate
BundleMetricsPlugin\r\n- [x] Check if the profiling plugins still work
(--profile flag)\r\n- [x] Recover if possible the previous webpack v4
cacheGroup chunks\r\nrename to something like
`data.plugin.chunk.0.js`\r\n- [x] Run `/ci` and make sure we get our
first green CI, otherwise work\r\non the errors until we do\r\n- [x]
Profile and solve bottlenecks until we get a cold build\r\nperformance
similar to the one we had on webpack v4
(`node\r\nscripts/build_kibana_platform_plugins --no-cache`).\r\n- [x]
OpenSSL Legacy Warnings: try to remove `--openssl-legacy-provider\r\n`
flags\r\n- [x] Add Webpack to Renovate config\r\n- [x] Explore removing
`NodePolyfillPlugin`\r\n([here](https://www.npmjs.com/package/node-polyfill-webpack-plugin))
and\r\nadd each polyfill needed individually per each webpack config to
check\r\nif we get smaller bundles. If we do it's better to go with the
case by\r\ncase need approach instead of deploying a bunch of polyfills
with\r\nNodePolyfillPlugin. As another alternative, create a custom
smaller\r\nplugin with only the union of all needed polyfills.\r\n- [x]
Evaluate if we want to touch the resolutions on mainFields
and\r\nconditionNames\r\n- [x] Understand why `@import
'src/core/public/mixins'` does not work\r\nanymore (not a problem, we
should use relative paths anyway but we want\r\nto track why it changed
from v4 to v5)\r\n- [x] BUG: Child compilers are having errors hidden
and/or changed from\r\nerror to warning\r\n- [x] Fix license check
for\r\n[Artistic-2.0](https://spdx.org/licenses/Artistic-2.0.html) is
the\r\nlicense
for\r\n[domain-browser](https://github.com/bevry/domain-browser?tab=License-1-ov-file).\r\nThis
package is a dependency
of\r\n[NodePolyfillPlugin](https://www.npmjs.com/package/node-polyfill-webpack-plugin).\r\nArtistic
2.0 license is [classified
as\r\nyellow](https://github.com/elastic/open-source/blob/main/elastic-product-policy.md#yellow-list)\r\nand
should only be used for dev dependencies.\r\n- [x] Make sure
`resourceQuery: { not: /raw/ }` is not necessary on\r\nother webpack
configs like storybook one\r\n- [x] Find what is being wrongly removed
by usedExports optimization;\r\nhint: I believe it is identifying a lot
of exports inside the sync entry\r\nof plugins as unused exports and
removing them. Then `__kbnBootstrap__`\r\ncan't be found\r\n- [x]
Rebalance @kbn/optimizer pickMaxWorkerCount\r\n- [x] Re-open the issue
to fix
sass-warnings\r\n[#190345](https://github.com/elastic/kibana/issues/190345)
or downgrade\r\nsass-loader to v10\r\n- [x] Remove previous esm no parse
rules\r\n- [x] Confirm esm support is working\r\n- [x] Confirm console
override is needed\r\n- [x] Confirm react prod builds on ui shared deps
for distributable\r\n- [x] Remove customization
for\r\n[xyflow](https://github.com/xyflow/xyflow) from webpack
configs\r\n- [x] Clean all the code\r\n- [x] Make sure collected metrics
from stats are still aligned with what\r\nwe were collecting before;
also verify if the modules used for optimizer\r\ncaches etc are well
generated (@kbn/node-libs-browser)\r\n- [x] Fix watch
performance\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by:
Brad White
<brad.white@elastic.co>","sha":"203bc284781cc830049b3f7948c8fcafa8b7d9a1"}},{"url":"https://github.com/elastic/kibana/pull/211146","number":211146,"branch":"8.x","state":"OPEN"},{"url":"https://github.com/elastic/kibana/pull/211147","number":211147,"branch":"8.18","state":"OPEN"},{"url":"https://github.com/elastic/kibana/pull/211148","number":211148,"branch":"8.17","state":"OPEN"}]}]
BACKPORT-->

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2025-02-19 04:15:36 +00:00
Tiago Costa
fe88bea5a4
fix(NA): jest integration snapshots for packages/kbn-plugin-helpers/src/integration_tests/build.test.ts 2024-10-17 03:44:12 +01:00
Luke Elmers
b6287708f6
Adds AGPL 3.0 license (#192025)
Updates files outside of x-pack to be triple-licensed under Elastic
License 2.0, AGPL 3.0, or SSPL 1.0.
2024-09-06 19:02:41 -06:00
Alejandro Fernández Haro
7dfe90eeca
[Move @kbn/config-schema to server] @kbn/optimizer (#189769) 2024-08-02 13:04:23 -05:00
Pierre Gayvallet
7f3f757a38
[i18n] include i18nrc file in 3rd party plugin bundles (#188814)
## Summary

Fix #57273

Include the `. i18nrc.json` file when bundling 3rd party plugins
2024-07-22 15:40:35 +02:00
Tiago Costa
aef197b243
fix(NA): do not change default encoding when using vfs to copy files (#181174)
Closes https://github.com/elastic/kibana/issues/180067

When we upgraded into vinyl-fs@4.0.0 there was a missing breaking change
into their changelog related to encodings that was only added last
february. Because of that we missed a couple places where file
encondings were being changed from the default ones and in some cases
corrupting those files.

This PR fixes those problems.
2024-04-18 18:35:47 +01:00
Tiago Costa
891d79b9e7
fix(NA): skip plugin-helpers assets optimization when no ui (#173703)
This fixes a bug where the asset optimization script will try to run
even when we don't have any ui plugin generated.
2023-12-20 14:44:33 +00:00
Tiago Costa
ecdd1f090b
Remove webpack compression plugin on v4 (#170203)
Closes https://github.com/elastic/kibana/issues/170202

This PR fixes a problem with brotli generated assets when creating an
external plugin by replacing the brotli assets generated by webpack with
the ones created separately with gulp brotli.
2023-11-02 17:11:48 +00:00
Tiago Costa
c3936571fc
fix(NA): missing --openssl-legacy-provider on webpack worker inside plugin helpers (#169030)
This PR fixes a problem inside plugin-helpers as the webpack runner was
running without `--openssl-legacy-provider` which breaks in the latest
node version.
2023-10-16 23:09:52 +01:00
Jon
aebd6f3923
[node] Enable openssl legacy provider (#163190)
This is to prevent a breaking change in a minor release of Kibana due to
an underlying upgrade of Node.js to v18.
The legacy provider can be disabled by removing
`--openssl-legacy-provider` in `config/node.options`.

[Node.js
documentation](https://nodejs.org/docs/latest-v18.x/api/cli.html#--openssl-legacy-provider)
[OpenSSL
documentation](https://www.openssl.org/docs/man3.0/man7/OSSL_PROVIDER-legacy.html)

---------

Co-authored-by: Thomas Watson <w@tson.dk>
2023-08-30 11:51:10 -07:00
Alex Szabo
8cf68dc6ba
[Ops] Bump Node.js to version 18 (#160289)
## Summary

Bumps node.js to 18.17.0 (replacement for PR #144012 which was later
reverted)

As a result, these categorical additions were needed: 
- `node` evocations will need the `--openssl-legacy-provider` flag,
wherever it would use certain crypto functionalities
- tests required updating of the expected HTTPS Agent call arguments,
`noDelay` seems to be a default
 - `window.[NAME]` fields cannot be written directly
 - some stricter typechecks

This is using our in-house built node.js 18 versions through the URLs
the proxy-cache. (built with
https://github.com/elastic/kibana-custom-nodejs-builds/pull/4)

These urls are served from a bucket, where the RHEL7/Centos7 compatible
node distributables are. (see:
https://github.com/elastic/kibana-ci-proxy-cache/pull/7)

Further todos: 
 - [x] check docs wording and consistency
 - [ ] update the dependency report
 - [x] explain custom builds in documentation
 - [x] node_sass prebuilts

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
Co-authored-by: Thomas Watson <w@tson.dk>
2023-07-27 14:12:48 +02:00
Tiago Costa
7e067ec478
fix(NA): build external plugins with correct bazel artifacts output mode (#159630)
Closes https://github.com/elastic/kibana/issues/159328 ,
https://github.com/elastic/kibana/issues/154325,
https://github.com/elastic/kibana/issues/157440

This PR fixes a problem in the plugin helpers tasks used to build
external plugins. After changes to transpile packages differently in the
past months a bug was introduced and external plugins were being built
with the wrong shared ui deps output mode causing development form
imports to be bundled and causing problems at page load time.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-06-15 00:43:05 +01:00
Tiago Costa
66ac756b98
fix(NA): external plugins development flow with the new modern package plugins in place (#153562)
This PR fixes the 3rd party external plugin development workflow by
introducing a dev step for plugins that allows for development usages of
those with Kibana.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-04-06 18:00:24 +01:00
Spencer
1b85815402
[packages] migrate all plugins to packages (#148130)
Fixes https://github.com/elastic/kibana/issues/149344

This PR migrates all plugins to packages automatically. It does this
using `node scripts/lint_packages` to automatically migrate
`kibana.json` files to `kibana.jsonc` files. By doing this automatically
we can simplify many build and testing procedures to only support
packages, and not both "packages" and "synthetic packages" (basically
pointers to plugins).

The majority of changes are in operations related code, so we'll be
having operations review this before marking it ready for review. The
vast majority of the code owners are simply pinged because we deleted
all `kibana.json` files and replaced them with `kibana.jsonc` files, so
we plan on leaving the PR ready-for-review for about 24 hours before
merging (after feature freeze), assuming we don't have any blockers
(especially from @elastic/kibana-core since there are a few core
specific changes, though the majority were handled in #149370).

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-02-08 21:06:50 -06:00
Spencer
afb09ccf8a
Transpile packages on demand, validate all TS projects (#146212)
## Dearest Reviewers 👋 

I've been working on this branch with @mistic and @tylersmalley and
we're really confident in these changes. Additionally, this changes code
in nearly every package in the repo so we don't plan to wait for reviews
to get in before merging this. If you'd like to have a concern
addressed, please feel free to leave a review, but assuming that nobody
raises a blocker in the next 24 hours we plan to merge this EOD pacific
tomorrow, 12/22.

We'll be paying close attention to any issues this causes after merging
and work on getting those fixed ASAP. 🚀

---

The operations team is not confident that we'll have the time to achieve
what we originally set out to accomplish by moving to Bazel with the
time and resources we have available. We have also bought ourselves some
headroom with improvements to babel-register, optimizer caching, and
typescript project structure.

In order to make sure we deliver packages as quickly as possible (many
teams really want them), with a usable and familiar developer
experience, this PR removes Bazel for building packages in favor of
using the same JIT transpilation we use for plugins.

Additionally, packages now use `kbn_references` (again, just copying the
dx from plugins to packages).

Because of the complex relationships between packages/plugins and in
order to prepare ourselves for automatic dependency detection tools we
plan to use in the future, this PR also introduces a "TS Project Linter"
which will validate that every tsconfig.json file meets a few
requirements:

1. the chain of base config files extended by each config includes
`tsconfig.base.json` and not `tsconfig.json`
1. the `include` config is used, and not `files`
2. the `exclude` config includes `target/**/*`
3. the `outDir` compiler option is specified as `target/types`
1. none of these compiler options are specified: `declaration`,
`declarationMap`, `emitDeclarationOnly`, `skipLibCheck`, `target`,
`paths`

4. all references to other packages/plugins use their pkg id, ie:
	
	```js
    // valid
    {
      "kbn_references": ["@kbn/core"]
    }
    // not valid
    {
      "kbn_references": [{ "path": "../../../src/core/tsconfig.json" }]
    }
    ```

5. only packages/plugins which are imported somewhere in the ts code are
listed in `kbn_references`

This linter is not only validating all of the tsconfig.json files, but
it also will fix these config files to deal with just about any
violation that can be produced. Just run `node scripts/ts_project_linter
--fix` locally to apply these fixes, or let CI take care of
automatically fixing things and pushing the changes to your PR.

> **Example:** [`64e93e5`
(#146212)](64e93e5806)
When I merged main into my PR it included a change which removed the
`@kbn/core-injected-metadata-browser` package. After resolving the
conflicts I missed a few tsconfig files which included references to the
now removed package. The TS Project Linter identified that these
references were removed from the code and pushed a change to the PR to
remove them from the tsconfig.json files.

## No bazel? Does that mean no packages??
Nope! We're still doing packages but we're pretty sure now that we won't
be using Bazel to accomplish the 'distributed caching' and 'change-based
tasks' portions of the packages project.

This PR actually makes packages much easier to work with and will be
followed up with the bundling benefits described by the original
packages RFC. Then we'll work on documentation and advocacy for using
packages for any and all new code.

We're pretty confident that implementing distributed caching and
change-based tasks will be necessary in the future, but because of
recent improvements in the repo we think we can live without them for
**at least** a year.

## Wait, there are still BUILD.bazel files in the repo
Yes, there are still three webpack bundles which are built by Bazel: the
`@kbn/ui-shared-deps-npm` DLL, `@kbn/ui-shared-deps-src` externals, and
the `@kbn/monaco` workers. These three webpack bundles are still created
during bootstrap and remotely cached using bazel. The next phase of this
project is to figure out how to get the package bundling features
described in the RFC with the current optimizer, and we expect these
bundles to go away then. Until then any package that is used in those
three bundles still needs to have a BUILD.bazel file so that they can be
referenced by the remaining webpack builds.

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2022-12-22 19:00:29 -06:00
spalger
52f2b33a07
[auto] migrate existing plugin/package configs 2022-10-28 14:06:46 -05:00
spalger
42879f7656
[bazel] fix some BUILD.bazel file inconsistencies 2022-10-26 11:07:55 -05:00
Spencer
32491462a9
add kibana.jsonc files to existing packages (#138965)
* [packages] add kibana.jsonc files

* auto-migrate to kibana.jsonc

* support interactive pkg id selection too

* remove old codeowners entry

* skip codeowners generation when .github/CODEOWNERS doesn't exist

* fall back to format validation if user is offline

* update question style

* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2022-09-08 13:31:57 -07:00
Tiago Costa
1cbf83f73c
chore(NA): remove src folder requirement from packages (part 2) (#138476)
* refact(NA): apply root_input_dir=src to each already created pkg

* refact(NA): update package generator

* fix(NA): correctly use rootDir

* fix(NA): use root input dir on latest introduced pkgs for jsts_transpiler macro

* chore(NA): merge with main

* chore(NA): first attempt to complete removal of src folder on a small group of pkgs

* Revert "chore(NA): first attempt to complete removal of src folder on a small group of pkgs"

This reverts commit b6f34b7530.

* chore(NA): remove src folder requirement from xpack pkgs

* chore(NA): remove src folder from analytics pkgs

* chore(NA): remove src folder from home pkgs

* chore(NA): remove src folder from shared_ux pkgs

* fix(NA): remove missing src folder inputs on sharedux pkg

* chore(NA): remove src folder from kbn-a* pkgs

* chore(NA): remove src folder from kbn-b* pkgs

* chore(NA): remove src folder from kbn-c* pkgs

* chore(NA): correct exclude pattern for each changed pkg

* chore(NA): remove src folder from kbn-y* pkgs

* chore(NA): remove src folder from kbn-e* pkgs

* chore(NA): remove src folder from kbn-f* and kbn-g* pkgs

* chore(NA): remove src folder from kbn-f* and kbn-g* pkgs

* chore(NA): remove src folder from kbn-h** pkgs

* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'

* Revert "chore(NA): remove src folder from kbn-h** pkgs"

This reverts commit dcdf72bcc1.

* fix(NA): grammar location

* test(NA): fix tests for kbn/config-schema

* test(NA): fix tests for kbn/config-schema

* chore(NA): multiple errors fixed

* chore(NA): remove kuery grammar fix

* fix(NA): @kbn/ace imports

* fix(NA): grammar location

* fix(NA): add missing files to tsconfigs

* [CI] Auto-commit changed files from 'node scripts/build_plugin_list_docs'

* chore(NA): complete tsconfigs

* Revert "chore(NA): complete tsconfigs"

This reverts commit f48c616864.

* chore(NA): remove src folder from kbn-core* pkgs

* chore(NA): remove src folder from kbn-u* pkgs

* chore(NA): remove src folder from kbn-ui-shared-deps* pkgs

* chore(NA): fix problems on core pkgs

* chore(NA): fix problems on core pkgs

* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'

* fix(NA): shared_built_assets correct location'

* test(NA): update @kbn/optimizer integration snapshots

* chore(NA): remove src folder from kbn-t* p1 pkgs

* chore(NA): remove src folder from kbn-t* p2 pkgs

* chore(NA): update rootDir on utility types pkg

* chore(NA): include missing files on @kbn/test

* chore(NA): include missing files on @kbn/test

* fix(NA): new jest preset paths on @kbn/test

* chore(NA): fix eslint

* fix(NA): new jest preset paths on @kbn/test

* chore(NA): remove exclusion for mocks folde on @kbn/test

* fix(NA): several imports from target_node

* fix(NA): @kbn/test mocha and jest types clash

* chore(NA): remove src folder from kbn-storybook pkg

* chore(NA): remove src folder from kbn-stdio-dev-helpers pkg

* chore(NA): remove src folder from kbn-std pkg

* chore(NA): remove src folder from kbn-sort-pkg-json pkg

* chore(NA): remove src folder from kbn-some-dev-log and kbn-shared-ux-utility pkgs

* chore(NA): remove src folder from kbn-ux-storybook pkg

* chore(NA): remove src folder from kbn-shared-ux-services pkg

* chore(NA): remove src folder from kbn-shared-ux-components pkg

* chore(NA): remove src folder from kbn-shared-svg pkg

* chore(NA): remove src folder from kbn-server-http-tools pkg

* chore(NA): remove src folder from kbn-securitysolution-* pkgs

* chore(NA): remove src folder from kbn-r-* pkgs

* chore(NA): remove src folder from kbn-p* pkgs

* chore(NA): remove src folder from kbn-o* pkgs

* chore(NA): remove src folder from kbn-m* pkgs

* chore(NA): remove src folder from kbn-j,k,l* pkgs

* chore(NA): remove src folder from kbn-j,k,l* pkgs

* chore(NA): remove src folder from kbn-io-ts-utils* pkgs

* chore(NA): remove src folder from kbn-* pkgs except a few

* chore(NA): update @kbn/generate

* fix(NA): wrong exclusion on kbn-storybook

* chore(NA): remove src folder from kbn-monaco pkg

* chore(NA): remove src folder from kbn-interpreter pkg

* fix(NA): wrong exclusion on kbn-storybook

* chore(NA): update every require for target_*/src

* chore(NA): remover src folder from @kbn/handlebars

* fix(NA): license for @kbn/handlebars

* chore(NA): copy templates as part of the jsts_transpiler macro for @kbn/storybook

* chore(NA): update handlebars

* fix(NA): @kbn/plugin-generator import paths

* fix(NA): bundle sizes

* fix(NA): web bundle for @kbn/i18n-react

* Revert "fix(NA): bundle sizes"

This reverts commit 8aefe84fbc.

* Revert "Revert "fix(NA): bundle sizes""

This reverts commit e9d87d72a4.

* fix(NA): @kbn/docs-utils index.ts path expectation

* chore(NA): merge and solve conflicts with main

* fix(NA): relative import to index

* chore(NA): merge and solve conflicts with main

* [CI] Auto-commit changed files from 'node scripts/precommit_hook.js --ref HEAD~1..HEAD --fix'

* chore(NA): apply eslint fix

* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2022-08-30 15:57:35 +01:00
Jonathan Budzenski
c4da7bd75a
[build] Remove gzip files (#138508)
* [build] Remove gzip files

All browsers in our support matrix support brotli.
https://caniuse.com/brotli

We still have fallbacks to hapi gzip, and uncompressed in the event of
an error.

* update snapshots

* fix

* update snapshots
2022-08-17 08:54:14 -05:00
Tiago Costa
df8b62aa98
chore(NA): allow packages to not have an obligation for a src folder on packages (#138038)
* refact(NA): apply root_input_dir=src to each already created pkg

* refact(NA): update package generator

* fix(NA): correctly use rootDir

* fix(NA): use root input dir on latest introduced pkgs for jsts_transpiler macro

* chore(NA): merge with main

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2022-08-08 15:54:05 +01:00
Spencer
a3b2757e4e
[type-summarizer] reimplement for broader support (#135163)
* [type-summarizer] reimplement for broader support

* Enable sourceMaps in all packages

* include naming collision in summarizePackage test

* fix readmes

* remove unnecessary transient dependency

* remove code that was commented out

* remove outdated todo comment

* ensure errors triggered by untyped-exports are ligible

* remove unused import

* break out snippet generation from AstIndexer

* refactor several massive files into smaller pieces and add more inline docs

* fix typos

* update jest snapshots

* add sections to readme that points people to the useful parts of the source code along with a high-level overview of how the type-summarizer works

* remove --dump flag, it doesn't work

* use decName instead of calling names.get a second time

* include `export` as invalid name
2022-07-06 13:48:45 -05:00
Spencer
f3d69b8197
[@kbn/dev-utils] break out more pieces (#132292)
* [@kbn/dev-utils] break out more pieces

* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2022-05-17 11:19:20 -05:00
Spencer
2a78f350e1
break out parts of @kbn/dev-utils (#130509)
* break out parts of @kbn/dev-utils

* autofix imports and kbn/pm dist

* update readme for @kbn/stdio-dev-helpers

* finish renames
2022-04-19 12:24:58 -05:00
Spencer
92d65484a5
[plugin-discovery] move logic to a package (#128684) 2022-03-29 13:19:58 -06:00
Jonathan Budzenski
371c52a1ed
[plugin-helpers] Skip build.test.ts log matching on ci-stats service (#127560)
This copies the fix over from
https://github.com/elastic/kibana/pull/123510 to be used in both
plugin-helpers build calls.

Closes #89079
2022-03-14 09:06:26 -05:00
Spencer
4f2321d34c
[ToolingLog] remove some manual indent management (#126636)
* [ToolingLog] remove some manual indent management

* improve log message

* [dev/build] write the newline after a failed step

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2022-03-03 08:15:46 -07:00
Spencer
614139b8e5
[bazel] avoid a little boilerplate in packages (#126309)
* [bazel] avoid a little boilerplate for @types packages

* [bazel/ts] stop building sourcemaps since they're ignored
2022-02-24 12:43:12 -06:00
Spencer
1a1a191eb7
[jest] add *_node presets (#126192) 2022-02-23 09:36:30 -06:00
Tyler Smalley
435b772786
[ci] Splits Jest integration tests (#125454) 2022-02-14 17:12:42 +01:00
Spencer
72559010f7
[plugin-helpers/tests] filter out ci-stats warnings (#123510) 2022-01-20 15:19:13 -07:00
Tiago Costa
64468b7a3e
chore(NA): splits types from code on @kbn/utils (#121924) 2021-12-23 03:18:10 +00:00
Tiago Costa
a2d7a98353
chore(NA): splits types from code on @kbn/plugin-helpers (#121513)
* chore(NA): splits types from code on @kbn/monaco

* chore(NA): splits types from code on @kbn/plugin-helpers
2021-12-17 12:56:32 -07:00
Tiago Costa
ecf2265d56
chore(NA): splits types from code on @kbn/optimizer (#121426) 2021-12-16 20:04:20 +00:00
Tiago Costa
de5dd787b1
chore(NA): splits types from code on @kbn/dev-utils (#120531)
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-12-07 20:12:18 +00:00
Tyler Smalley
f1e9668aa2
[npm] Updates Browserslist caniuse-lite database (#112523)
* Updates Browserslist caniuse-lite database

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

* Revert "skip flaky suite"

This reverts commit 2fd91a4f07.
2021-09-19 07:50:53 -07:00
Tyler Smalley
2fd91a4f07 skip flaky suite
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
2021-09-18 21:21:39 -07:00
Tiago Costa
7c4e4f507b
chore(NA): replace babel config files by inline presets by default in the jsts_transpiler rule (#110620)
* chore(NA): replace babel config files by inline presets by default in the js_ts_transpiler rule

* chore(NA): update @kbn/ace build to exclude worker file

* chore(NA): remove config file support

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-09-03 21:17:46 +01:00
Stacey Gammon
52ee65b8d5
Make owner attribute required on kibana.json (#108231)
* make owner attribute required

* Add owner properties in more places

* add test for owner attribute

* add error check too in the test

* Fix tests

* fix tests and update docs

* wip

* More test fixes

* Fix All The Errorz

* Adding more owner attributes

* Update x-pack/test/saved_object_api_integration/common/fixtures/saved_object_test_plugin/kibana.json

Co-authored-by: Larry Gregory <lgregorydev@gmail.com>

* Update x-pack/test/ui_capabilities/common/fixtures/plugins/foo_plugin/kibana.json

Co-authored-by: Larry Gregory <lgregorydev@gmail.com>

* commeeeooonnnn

* Update docs

* soooo many kibanajsons

* adjust plugin generator to add an owner

* Add owner to the plugin generator scripts

* update snapshot

* Fix snapshot

* review updates

Co-authored-by: Larry Gregory <lgregorydev@gmail.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-08-24 10:02:32 -04:00
Tiago Costa
a7fe773bb8
chore(NA): moving @kbn/plugin-helpers to babel transpiler (#109085)
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-08-19 19:13:33 +01:00
Spencer
c0395c9ef6
[build_ts_refs] improve caches, allow building a subset of projects (#107981)
* [build_ts_refs] improve caches, allow building a subset of projects

* cleanup project def script and update refs in type check script

* rename browser_bazel config to avoid kebab-case

* remove execInProjects() helper

* list references for tsconfig.types.json for api-extractor workload

* disable composite features of tsconfig.types.json for api-extractor

* set declaration: true to avoid weird debug error

* fix jest tests

Co-authored-by: spalger <spalger@users.noreply.github.com>
2021-08-10 22:12:45 -07:00
Jonathan Budzenski
0b7eb3f85f
[plugin-helpers] Fix dependency resolution (#104556)
This builds on the changes introduced in #104512, fixing dependency
resolution for the plugin-helpers CLI.  Instead of building a plugin in
isolation, this performs plugin discovery and filters the optimizer
lisst down to the requested plugin.

Testing
1) `cd examples/bfetch_explorer`
1) `node ../../scripts/plugin_helpers build --kibana-version=8.0.0-SNAPSHOT`
2021-07-07 12:40:58 -05:00
Tiago Costa
59423927cb
chore(NA): moving @kbn/plugin-helpers into bazel (#103181)
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-06-23 23:49:04 -04:00
Tiago Costa
e1ec8b05b6
chore(NA): moving @kbn/optimizer into bazel (#102965)
* chore(NA): moving @kbn/optimizer into bazel

* chore(NA): fix source import from kbn optimizer

* chore(NA): update snapshots
2021-06-23 11:11:13 +01:00
Tiago Costa
f690c60517
chore(NA): moving @kbn/dev-utils into bazel (#98496)
* chore(NA): moving @kbn/dev-utils into bazel

* docs(NA): updated generated plugin list
2021-04-28 23:46:33 +01:00
Tiago Costa
50313f75f6
chore(NA): enable preserve symlinks for ts without breaking packages development (#95433)
* chore(NA): move elastic-datemath into a ts package

* chore(NA): finish elastic-datemath

* chore(NA): finish elastic-datemath

* chore(NA): source folder for elastic-datemath

* chore(NA): add source-maps ace, analytics, apm-config-loader and apm-utils packages

* chore(NA): add sourcemaps to packages on typescript

* chore(NA): move test fixtures within source

* chore(NA): correct exclusions on packages

* chore(NA): correct package.json on all packages

* chore(NA): correct package.json on all packages

* chore(NA): complete kbn pm

* chore(NA): default export on elastic-datemath

* chore(NA): include logs on kbn-logging

* chore(NA): update bundle ref module to last code used in the webpack upstream

* chore(NA): update bundle ref module to last code used in the webpack upstream - refactored

* chore(NA): remove override method for exportsArgument

* fix(NA): typechecking problems by use @internal at javascript import sources on kbn-test package

* fix(NA): typescript projects check

* fix(NA): run optimizer integration tests from source

* chore(NA): fix usage from target for kbn optimizer

* chore(NA): path on tsconfig

* chore(NA): move tsignore into ts-expect-error

* chore(NA): include souce maps on kbn cli dev

* chore(NA): include souce maps on kbn-crypto, kbn-server-http-tools and kbn-telemetry-tools

* chore(NA): add issue links into the ts-expect-error comments
2021-03-31 00:02:22 +01:00
Spencer
a0ce7b5aa8
[kbn/optimizer][ci-stats] ship metrics separate from build (#90482)
Co-authored-by: spalger <spalger@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-02-08 10:47:55 -07:00
Brandon Kobel
4584a8b570
Elastic License 2.0 (#90099)
* Updating everything except the license headers themselves

* Applying ESLint rules

* Manually replacing the stragglers
2021-02-03 18:12:39 -08:00
Brandon Kobel
170a2956c8
Updating the License (#88343)
* Updating the Licenses, except for applying eslint, building

* Applying ESLint rules,building @kbn/pm, regenerating api docs
2021-01-19 17:52:56 -08:00
Spencer
859ba3efa3
[kbn/plugin-helpers/build] copy the public assets of a plugin (#83458)
Co-authored-by: spalger <spalger@users.noreply.github.com>
2020-11-16 15:07:57 -07:00