github-mirrors/kibana
1
0
Fork 0
mirror of https://github.com/elastic/kibana.git synced 2025-04-19 15:35:00 -04:00
Code Issues Projects Releases Packages Wiki Activity

Create vendor dll for the client modules (#22618)

Browse source 
* feat(NA): first dll bundler code.

* chore(NA): upgrade to webpack 4.

* chore(NA): updated package.json

* chore(NA): removed old code.

* chore(NA): add parallel option.

* chore(NA): removed console log and old var about  node modules.

* chore(NA): turn off unsafe cache.

* chore(NA): update lock files.

* chore(NA): new config for dll generation.

* chore(NA): update stats emit.

* chore(NA): update dependencies.

* chore(NA): right cache loaders.

* chore(NA): remove ui_bundles alias.

* feat(20749): init implementation on bridge plugin for dll bundler.

* feat(20749): init implementation for dll compiler.

* feat(20749): dll bundler init from other process and webpack wrapper..

* feat(20749): optimizer changes to integrate with  dll bundler.

* chore(20749): split into different processes.

* refact(20749): change to single running process.

* refact(NA): improvements on dll bundler plugin.

* refact(NA): removing including loop on plugins.

* refact(20749): only run dllReference once.

* chore(20749): todo on result.request path building.

* chore(NA): lock files updated.

* chore(NA): avoiding dll paths being removed.

* chore(NA): tests on sync generation.

* chore(NA): changes on entry paths compiler.

* chore(NA): different hooks tests.

* chore(20749): first working version, single process, for dynamic building dll.

* feat(20749): last gross features for the dynamicdllplugin.

* refact(20749): string interpolation when creating the path to delete in every optimizer cycle. feat(20749): creating the DynamicDllPlugin foundations.

* chore(NA): updated base optimizer run function args.

* chore(20749): first working solution with vendor dll both for prod and dev.

* chore(20749): useful todos on client dlls.

* feat(NA): auto built blacklist for server node_modules.

* refact(NA): removed unused code.

* chore(NA): update all webpack and loaders stuff to last versions.

* refact(NA): first refacts on clean taks related with dll. feat(NA): added clean empty folders task.

* refact(NA): removed support for console logs during the build.

* refact(NA): removed extra space.

* refact(NA): removed extra space.

* refact(NA): getDllModules function.

* chore(NA): removed unsafeCache option.

* feat(NA): removed unsafeCache option.

* refact(NA): apply general inheritance principles to the optimizer: hooks, and init. refact(NA): merge dynamic dll plugin into the common config. refact(NA): restore old template structure - vendors.dll style is always emitted.

* fix(NA): fs_optimizer run function by not returning inside async func.

* fix(NA): change template anchor name from vendor to vendors.

* docs(NA): added info about files were keeping when cleaning the bundles.

* fix(NA): filtering elible dll modules to delete.

* refact(NA): removed old dll bundler in favor on new dynamic dll plugin.

* refact(NA): initial split for clean modules on dll task.

* refact(NA): update new dll config model. refact(NA): update config on dynamicdllplugin.

* refact(NA): major work refactor for dynamic dll plugin.

* refact(NA): extract clean node modules on dll task to its own folder.

* refact(NA): organize imports.

* docs(NA): add docs to registerCompilerHooks function for the optimizer.

* refact(NA): finished refactor for dynamic dll plugin with correct error handling for runWebpack function.

* refact(NA): basic structure for clean client modules on dll task.

* fix(NA): resolve path for dll manifest during cclean build tasks.

* refact(NA): split webpack dll related functions to their own file for clean client modules on dll task.

* refact(NA): added error handling for the clean client modules on dll task - webpack dll related functions.

* docs(NA): added license header.

* refact(NA): complete split out the functions from the clean modules on dll task to the code_parser file.

* refact(NA): main task entries compose.

* docs(NA): extend docs for the getDependenciesFromFile function.

* refact(NA): final structure split for clean client node modules dll task.

* fix(NA): added missing param to calculate top level dependencies.

* docs(NA): completed todo description about dll location.

* fix(NA): add production option to webpack config on kbn-pm.

* docs(NA): extended documentation about style extraction.

* refact(NA): removed extra comments.

* feat(NA): env variable to force dll creation.

* feat(NA): include option to define folders to keep on delete empty folders task.

* refact(NA): remove unused method from dll compiler.

* feat(NA): move dlls outside bundles folder and support request for /dlls from a browser perspective.

* chore(NA): gitignore updated to include new dlls folder.

* chore(NA): eslintignore updated.

* chore(NA): removed strange file from repo.

* test(NA): fix failing tests for bundles_route.

* fix(NA): change paths array to path string in a server route config.

* fix(NA): remove infinite loop calls on register hooks functions.

* fix(NA): readFile should only override the file in case it not exists.

* refact(NA): removed dll compiler finish log on success with stats.

* fix(NA): dll compiler alias.

* fix(NA): dynamic dll plugin flow on needs compile.

* fix(NA): raw alias config.

* Revert "fix(NA): raw alias config."

This reverts commit ebb245a786.

* feat(NA): raw alias for moment on dll config model.

* fix(NA): removed uiBundles from base_optimizer call on dynamicdllplugin.

* chore(NA): decrease moment versions.

* chore(NA): temporary changes on dll compiler.

* fix(NA): majority of problems between dll approach, webpackshims and browser tests.

* fix(NA): webpackShims integration with client vendors dll. fix(NA): enable esm modules mutability for development. fix(NA): only clean dll contents from build when they belong to node_modules.

* fix(NA): fix for endless dll compilation loop.

* fix(NA): removed esm plugin and skipped test using wrong stub strategy.

* docs(NA): added some comments for the skipped test.

* feat(NA): considering requires inside webpackShims valid entry paths to add to the dll entry file.

* fix(NA): small fix for the max compilation logic.

* docs(NA): add small explanatory note.

* fix(NA): building requires results with relative requires found onn webpackShims.

* docs(NA): add small note for error handling on dll compiler.

* fix(NA): move precinct to prod dependencies.

* test(NA): testing running functional tests on production.

* fix(NA): restore tests run config for development flag. feat(NA): force dll creation with uiBundles is Dev flag.

* chore(NA): update dependencies.

* feat(NA): test update dll to completely match base optimizer one.

* fix(NA): removed ts.

* refact(NA): removed unused consts.

* fix(NA): set back transpile sacss task in place.

* fix(NA): fix resolve remoing ts and tsx.

* fix(NA): set back transpile sacss task in place.

* fix(NA): removing isDevmode from mustCompileDll.

* fix(NA): add search for import statements into the dependencies visitor.

* fix(NA): add dll suffix to vendors resource on ui bootstrap template.

* fix(NA): some configs for unrelated dll work projects.

* chore(NA): upgrade canvas plugins webpack build to webpack4.

* chore(NA): add shim for moment-duration-format.

* chore(NA): stup moment-duration-format into the moment webpackShim.

* chore(NA): setup moment-duration-format into the moment-timezone webpackShim.

* fix(NA): moment and moment-timezone webpackShims

* chore(NA): added moment and moment-timezone webpackShims to x-pack. fix(NA): revert changes on webpackShims for oss kibana.

* fix(NA): xpack webpackshims for moment.

* fix(NA): remove xpack webpakcshims for moment.

* fix(NA): webpakcshims for moment.

* fix(NA): remove every changes from webpackShims and xpack webpackShims.

* fix(NA): fix visitors to gather server dependencies resulting from export * from and export x, 'x' from.

* chore(NA): update some webpack related dependencies.

* fix(NA): in the dll the plugins need to have their own dependencies. It is the same for the ones into the tests relying on test against distributable.

* feat(NA): including test/plugin_functional plugins into the kbn-pm bootstrap tasks.

* fix(NA): wrong built yarn lock files.

* chore(NA): updated webpack related dependencies.

* feat(NA): add missing color for dynamic_dll_plugin logging tag.

* chore(NA): removed forgotten console.log.

* chore(NA): removed forgotten dead code.

* chore(NA): removed missing old comment.

* docs(NA): added missing notice for 2 tools we have relied on.

* refact(NA): added is to a boolean variable to keep the consistency inside the code parser strategies.

* fix(NA): update notice cli to exclude search inside dlls bundles. chore(NA): update notice file.

* feat(NA): use lodash matches in the code parser visitors logic.

* docs(NA): updated notice file related with the code parser visitors logic..

* docs(NA): added explanation for the process that decides if we should build a new dll or not.

* test(NA): added missing tests for some usefull parts of the code.

* refact(NA): split registerCompileHook function into small ones.

* chore(NA): uncomment scripts from tests.

* feat(NA): isolate code-parser in a kbn package

* fix(NA): missing relative dot into the cwd function.

* chore(NA): update all inter deps to match.

* fix(NA): rebuild the yarn locks and the package jsons based on master ones.

* fix(NA): move babel-code-parser to the prod deps.

* chore(NA): include build path instead of the base root path.

* refact(NA): integrate plugin_functional test plugins with workspaces.

* fix(NA): include missing license for plugin functional test plugins.

* fix(NA): always write posix paths into the dll entry file in order to make the dlls compilation working on windows too. chore(NA): improve error handling into dll compiler.

* fix(NA): revert wrong moved line from canvas.

* fix(NA): match ts-loader version between kibana and x-pack.

* fix(NA): sync dll compiler with base_optimizer.

* fix(NA): exclude kbn-interpreter from the dll.

* refact(NA): remove exclusion of kbn-interpretor from base_optimizer.

* chore(NA): add dlls folder to the yarn kbn clean script.

* fix(NA): missing utf8 input format encoding when reading a file to create the hash into the watch optimizer cahce.

* refact(NA): re-engineering to the dynamic_dll_plugin logs and lifecycle.

* fix(NA): update clean node modules task to search under legacy/core_plugins.

* fix(NA): fix build on windows with globby on cleaning dlls for the watch optimizer cache.

* docs(NA): update documentation for the clean client node modules build task.

* docs(NA): added extra comment to clarify the purpose for the built entrypoints.

* chore(NA): update clean client node_modules code to use posix path.

* feat(NA): add support for discovering server entries over the legacy plugins and the new plugins.
This commit is contained in:
Tiago Costa 2018-12-05 15:45:19 +00:00 • committed by GitHub
parent da34c80d75
commit d5b2c8eaf2
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
91 changed files with 13478 additions and 13052 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
.ci/packer_cache.sh
View file

@ -22,4 +22,5 @@ tar -cf "$HOME/.kibana/bootstrap_cache/master.tar" \
x-pack/plugins/*/node_modules \
x-pack/plugins/reporting/.chromium \
x-pack/plugins/reporting/.phantom \
test/plugin_functional/plugins/*/node_modules \
.es;

1
.eslintignore
View file

@ -5,6 +5,7 @@ bower_components
/.es
/plugins
/optimize
/dlls
/src/fixtures/vislib/mock_data
/src/ui/public/angular-bootstrap
/src/ui/public/flot-charts

1
.gitignore vendored
View file

@ -9,6 +9,7 @@ node_modules
!/src/dev/notice/__fixtures__/node_modules
trash
/optimize
/dlls
target
/build
.jruby

2
CONTRIBUTING.md
View file

@ -230,8 +230,6 @@ The `config/kibana.yml` file stores user configuration directives. Since this fi
#### Potential Optimization Pitfalls
In development mode, Kibana runs a customized version of [Webpack](http://webpack.github.io/) with some optimizations enabled to make building the browser bundles as fast as possible. These optimizations make the build process about 2x as fast for initial builds, and about 7x faster for rebuilds, but are labeled "unsafe" by Webpack because they can sometimes cause changes to go unnoticed by the compiler. If you experience any of the scenarios below either restart the dev server, or add `optimize.unsafeCache: false` to your `config/kibana.dev.yml` file to disable these optimizations completely.
- Webpack is trying to include a file in the bundle that I deleted and is now complaining about it is missing
- A module id that used to resolve to a single file now resolves to a directory, but webpack isn't adapting
- (if you discover other scenarios, please send a PR!)

3
NOTICE.txt
View file

@ -1,6 +1,9 @@
Kibana source code with Kibana X-Pack source code
Copyright 2012-2018 Elasticsearch B.V.
---
This product has relied on ASTExplorer that is licensed under MIT.
---
This product includes code that was extracted from angular-ui-bootstrap@0.13.1
which is available under an "MIT" license

56
package.json
View file

@ -27,6 +27,7 @@
"extraPatterns": [
"build",
"optimize",
"dlls",
".eslintcache"
]
}
@ -71,39 +72,41 @@
"url": "https://github.com/elastic/kibana.git"
},
"resolutions": {
"**/@types/node": "8.10.21",
"**/@types/node": "8.10.38",
"@types/react": "16.3.14"
},
"workspaces": {
"packages": [
"packages/*",
"x-pack",
"x-pack/plugins/*"
"x-pack/plugins/*",
"test/plugin_functional/plugins/*"
],
"nohoist": [
"**/@types/*",
"**/@types/*/**",
"**/grunt-*",
"**/grunt-*/**",
"x-pack/typescript"
"x-pack/typescript",
"kbn_tp_*/**"
]
},
"dependencies": {
"@elastic/datemath": "5.0.1",
"@elastic/eui": "5.3.0",
"@elastic/filesaver": "1.1.2",
"@elastic/good": "8.1.1-kibana1",
"@elastic/numeral": "2.3.2",
"@elastic/ui-ace": "0.2.3",
"@kbn/babel-code-parser": "1.0.0",
"@kbn/babel-preset": "1.0.0",
"@kbn/config-schema": "1.0.0",
"@elastic/datemath": "5.0.1",
"@kbn/es-query": "1.0.0",
"@kbn/i18n": "1.0.0",
"@kbn/interpreter": "1.0.0",
"@kbn/pm": "1.0.0",
"@kbn/test-subj-selector": "0.2.1",
"@kbn/ui-framework": "1.0.0",
"JSONStream": "1.1.1",
"abortcontroller-polyfill": "^1.1.9",
"angular": "1.6.9",
"angular-aria": "1.6.6",
@ -113,19 +116,19 @@
"angular-sanitize": "1.5.7",
"angular-sortable-view": "0.0.15",
"autoprefixer": "^9.1.0",
"babel-core": "6.21.0",
"babel-loader": "7.1.2",
"babel-polyfill": "6.20.0",
"babel-register": "6.18.0",
"babel-core": "6.26.3",
"babel-loader": "7.1.5",
"babel-polyfill": "6.26.0",
"babel-register": "6.26.0",
"bluebird": "2.9.34",
"boom": "^7.2.0",
"brace": "0.11.1",
"cache-loader": "1.0.3",
"cache-loader": "1.2.2",
"chalk": "^2.4.1",
"color": "1.0.3",
"commander": "2.8.1",
"compare-versions": "3.1.0",
"css-loader": "0.28.7",
"css-loader": "1.0.0",
"custom-event-polyfill": "^0.3.0",
"d3": "3.5.6",
"d3-cloud": "1.2.1",
@ -136,8 +139,7 @@
"encode-uri-query": "1.0.0",
"execa": "^0.10.0",
"expiry-js": "0.1.7",
"extract-text-webpack-plugin": "3.0.1",
"file-loader": "1.1.4",
"file-loader": "2.0.0",
"font-awesome": "4.4.0",
"getos": "^3.1.0",
"glob": "^7.1.2",
@ -162,10 +164,11 @@
"leaflet-vega": "^0.8.6",
"leaflet.heat": "0.2.0",
"less": "2.7.1",
"less-loader": "4.0.5",
"less-loader": "4.1.0",
"lodash": "npm:@elastic/lodash@3.10.1-kibana1",
"lru-cache": "4.1.1",
"markdown-it": "^8.4.1",
"mini-css-extract-plugin": "0.4.4",
"minimatch": "^3.0.4",
"mkdirp": "0.5.1",
"moment": "^2.20.1",
@ -177,7 +180,7 @@
"opn": "^5.4.0",
"oppsy": "^2.0.0",
"pegjs": "0.9.0",
"postcss-loader": "2.0.6",
"postcss-loader": "3.0.0",
"prop-types": "15.5.8",
"proxy-from-env": "1.0.0",
"pug": "^2.0.3",
@ -210,7 +213,7 @@
"semver": "^5.5.0",
"socket.io": "^2.1.1",
"stream-stream": "^1.2.6",
"style-loader": "0.19.0",
"style-loader": "0.23.1",
"tar": "2.2.0",
"tinygradient": "0.3.0",
"tinymath": "1.1.0",
@ -222,17 +225,17 @@
"type-detect": "^4.0.8",
"uglifyjs-webpack-plugin": "^1.2.7",
"ui-select": "0.19.6",
"url-loader": "0.5.9",
"url-loader": "1.1.2",
"uuid": "3.0.1",
"val-loader": "^1.1.0",
"val-loader": "^1.1.1",
"validate-npm-package-name": "2.2.2",
"vega-lib": "^3.3.1",
"vega-lite": "^2.4.0",
"vega-schema-url-parser": "1.0.0",
"vega-tooltip": "^0.9.14",
"vision": "^5.3.3",
"webpack": "3.6.0",
"webpack-merge": "4.1.0",
"webpack": "4.23.1",
"webpack-merge": "4.1.4",
"whatwg-fetch": "^2.0.3",
"wreck": "^14.0.2",
"x-pack": "7.0.0",
@ -249,7 +252,7 @@
"@kbn/plugin-generator": "1.0.0",
"@kbn/test": "1.0.0",
"@octokit/rest": "^15.10.0",
"@types/angular": "^1.6.50",
"@types/angular": "1.6.50",
"@types/angular-mocks": "^1.7.0",
"@types/babel-core": "^6.25.5",
"@types/bluebird": "^3.1.1",
@ -282,7 +285,7 @@
"@types/minimatch": "^2.0.29",
"@types/moment-timezone": "^0.5.8",
"@types/mustache": "^0.8.31",
"@types/node": "^8.10.20",
"@types/node": "^8.10.38",
"@types/opn": "^5.1.0",
"@types/podium": "^1.0.0",
"@types/prop-types": "^15.5.3",
@ -346,7 +349,7 @@
"has-ansi": "^3.0.0",
"image-diff": "1.6.0",
"intl-messageformat-parser": "^1.4.0",
"istanbul-instrumenter-loader": "3.0.0",
"istanbul-instrumenter-loader": "3.0.1",
"jest": "^23.5.0",
"jest-cli": "^23.5.0",
"jest-raw-loader": "^1.0.1",
@ -370,10 +373,11 @@
"murmurhash3js": "3.0.1",
"mutation-observer": "^1.0.3",
"nock": "8.0.0",
"node-sass": "^4.9.0",
"node-sass": "^4.9.4",
"normalize-path": "^3.0.0",
"pixelmatch": "4.0.2",
"postcss": "^7.0.2",
"pkg-up": "^2.0.0",
"postcss": "^7.0.5",
"prettier": "^1.14.3",
"proxyquire": "1.7.11",
"regenerate": "^1.4.0",
@ -384,7 +388,7 @@
"supertest-as-promised": "^4.0.2",
"tree-kill": "^1.1.0",
"ts-jest": "^23.1.4",
"ts-loader": "^3.5.0",
"ts-loader": "^5.2.2",
"ts-node": "^7.0.1",
"tslint": "^5.11.0",
"tslint-config-prettier": "^1.15.0",

2
packages/elastic-datemath/package.json
View file

@ -13,7 +13,7 @@
"devDependencies": {
"babel-cli": "^6.26.0",
"babel-plugin-add-module-exports": "^0.2.1",
"babel-preset-env": "^1.6.1"
"babel-preset-env": "^1.7.0"
},
"dependencies": {
"moment": "^2.13.0",

3
packages/kbn-babel-code-parser/.babelrc Normal file
View file

@ -0,0 +1,3 @@
{
"presets": ["@kbn/babel-preset/node_preset"]
}

19
packages/kbn-babel-code-parser/README.md Executable file
View file

@ -0,0 +1,19 @@
# @kbn/babel-code-parser
Simple abstraction over the `@babel/parser` and the `@babel/traverse` in order
to build a code parser on top.
We have two main functions `parseSingleFile` (sync and sync version) and the
`parseEntries` (only async version). The first one just parse one entry file
and the second one parses recursively all the files from a list of
start entry points.
Then we have `visitors` and `strategies`. The first ones are basically the
`visitors` to use into the ast from the `@babel/traverse`. They are the only
way to collect info when using the `parseSingleFile`. The `strategies` are
meant to be used with the `parseEntries` and configures the info we want
to collect from our parsed code. After each loop, one per entry file, the
`parseEntries` method will call the given `strategy` expecting that
`strategy` would call the desired visitors, assemble the important
information to collect and adds them to the final results.

26
packages/kbn-babel-code-parser/package.json Executable file
View file

@ -0,0 +1,26 @@
{
"name": "@kbn/babel-code-parser",
"description": "babel code parser for Kibana",
"private": true,
"version": "1.0.0",
"main": "./target/index.js",
"license": "Apache-2.0",
"repository": {
"type": "git",
"url": "https://github.com/elastic/kibana/tree/master/packages/kbn-babel-code-parser"
},
"scripts": {
"build": "babel src --out-dir target --quiet",
"kbn:bootstrap": "yarn build",
"kbn:watch": "yarn build --watch"
},
"devDependencies": {
"babel-cli": "^6.26.0"
},
"dependencies": {
"@kbn/babel-preset": "1.0.0",
"@babel/parser": "^7.1.3",
"@babel/traverse": "^7.1.4",
"lodash": "^4.17.11"
}
}

36
packages/kbn-babel-code-parser/src/can_require.js Normal file
View file

@ -0,0 +1,36 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
export function canRequire(cwd, entry) {
try {
// We will try to test if we can resolve
// this entry through the require.resolve
// setting as the start looking path the
// given cwd. Require.resolve will keep
// looking recursively as normal starting
// from that location.
return require.resolve(entry, {
paths: [
cwd
]
});
} catch (e) {
return false;
}
}

103
packages/kbn-babel-code-parser/src/code_parser.js Normal file
View file

@ -0,0 +1,103 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import { canRequire } from './can_require';
import { readFile, readFileSync } from 'fs';
import { extname } from 'path';
import { promisify } from 'util';
import * as parser from '@babel/parser';
import traverse from '@babel/traverse';
import * as babelParserOptions from '@kbn/babel-preset/common_babel_parser_options';
const read = promisify(readFile);
function _cannotParseFile(filePath) {
return extname(filePath) !== '.js';
}
function _parseAndTraverseFileContent(fileContent, visitorsGenerator) {
const results = [];
// Parse and get the code AST
// All the babel parser plugins
// were enabled
const ast = parser.parse(fileContent, babelParserOptions);
// Loop through the code AST with
// the defined visitors
traverse(ast, visitorsGenerator(results));
return results;
}
export async function parseSingleFile(filePath, visitorsGenerator) {
// Don't parse any other files than .js ones
if (_cannotParseFile(filePath)) {
return [];
}
// Read the file
const content = await read(filePath, { encoding: 'utf8' });
// return the results found on parse and traverse
// the file content with the given visitors
return _parseAndTraverseFileContent(content, visitorsGenerator);
}
export function parseSingleFileSync(filePath, visitorsGenerator) {
// Don't parse any other files than .js ones
if (_cannotParseFile(filePath)) {
return [];
}
// Read the file
const content = readFileSync(filePath, { encoding: 'utf8' });
// return the results found on parse and traverse
// the file content with the given visitors
return _parseAndTraverseFileContent(content, visitorsGenerator);
}
export async function parseEntries(cwd, entries, strategy, results, wasParsed = {}) {
// Assure that we always have a cwd
const sanitizedCwd = cwd || process.cwd();
// Test each entry against canRequire function
const entriesQueue = entries.map(entry => canRequire(sanitizedCwd, entry));
while(entriesQueue.length) {
// Get the first element in the queue as
// select it as our current entry to parse
const mainEntry = entriesQueue.shift();
// Avoid parse the current entry if it is not valid
// or it was already parsed
if (typeof mainEntry !== 'string' || wasParsed[mainEntry]) {
continue;
}
// Find new entries and adds them to the end of the queue
entriesQueue.push(...(await strategy(sanitizedCwd, parseSingleFile, mainEntry, wasParsed, results)));
// Mark the current main entry as already parsed
wasParsed[mainEntry] = true;
}
return results;
}

22
packages/kbn-babel-code-parser/src/index.js Normal file
View file

@ -0,0 +1,22 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
export { dependenciesParseStrategy } from './strategies';
export { dependenciesVisitorsGenerator } from './visitors';
export { parseSingleFile, parseSingleFileSync, parseEntries } from './code_parser';

94
packages/kbn-babel-code-parser/src/strategies.js Normal file
View file

@ -0,0 +1,94 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import { canRequire } from './can_require';
import { dependenciesVisitorsGenerator } from './visitors';
import { dirname, isAbsolute, resolve } from 'path';
export function _calculateTopLevelDependency(inputDep, outputDep = '') {
// The path separator will be always the forward slash
// as at this point we only have the found entries into
// the provided source code entries where we just use it
const pathSeparator = '/';
const depSplitPaths = inputDep.split(pathSeparator);
const firstPart = depSplitPaths.shift();
const outputDepFirstArgAppend = outputDep ? pathSeparator : '';
outputDep += `${outputDepFirstArgAppend}${firstPart}`;
// In case our dependency isn't started by @
// we are already done and we can return the
// dependency value we already have
if (firstPart.charAt(0) !== '@') {
return outputDep;
}
// Otherwise we need to keep constructing the dependency
// value because dependencies starting with @ points to
// folders of dependencies. For example, in case we found
// dependencies values with '@the-deps/a' and '@the-deps/a/b'
// we don't want to map it to '@the-deps' but also to @'the-deps/a'
// because inside '@the-deps' we can also have '@the-dep/b'
return _calculateTopLevelDependency(depSplitPaths.join(pathSeparator), outputDep);
}
export async function dependenciesParseStrategy(cwd, parseSingleFile, mainEntry, wasParsed, results) {
// Retrieve native nodeJS modules
const natives = process.binding('natives');
// Get dependencies from a single file and filter
// out node native modules from the result
const dependencies = (await parseSingleFile(mainEntry, dependenciesVisitorsGenerator))
.filter(dep => !natives[dep]);
// Return the list of all the new entries found into
// the current mainEntry that we could use to look for
// new dependencies
return dependencies.reduce((filteredEntries, entry) => {
const absEntryPath = resolve(cwd, dirname(mainEntry), entry);
const requiredPath = canRequire(cwd, absEntryPath);
const requiredRelativePath = canRequire(cwd, entry);
const isRelativeFile = !isAbsolute(entry);
const isNodeModuleDep = isRelativeFile && !requiredPath && requiredRelativePath;
const isNewEntry = isRelativeFile && requiredPath;
// If it is a node_module add it to the results and also
// add the resolved path for the node_module main file
// as an entry point to look for dependencies it was
// not already parsed
if (isNodeModuleDep) {
// Save the result as the top level dependency
results[_calculateTopLevelDependency(entry)] = true;
if (!wasParsed[requiredRelativePath]) {
filteredEntries.push(requiredRelativePath);
}
}
// If a new, not yet parsed, relative entry were found
// add it to the list of entries to be parsed
if (isNewEntry && !wasParsed[requiredPath]) {
if (!wasParsed[requiredPath]) {
filteredEntries.push(requiredPath);
}
}
return filteredEntries;
}, []);
}

96
packages/kbn-babel-code-parser/src/strategies.test.js Normal file
View file

@ -0,0 +1,96 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import { readFile } from 'fs';
import { canRequire } from './can_require';
import { parseSingleFile } from './code_parser';
import { _calculateTopLevelDependency, dependenciesParseStrategy } from './strategies';
jest.mock('./can_require', () => ({
canRequire: jest.fn()
}));
jest.mock('fs', () => ({
readFile: jest.fn()
}));
const mockCwd = '/tmp/project/dir/';
describe('Code Parser Strategies', () => {
it('should calculate the top level dependencies correctly', () => {
const plainDep = 'dep1/file';
const foldedDep = '@kbn/es/file';
const otherFoldedDep = '@kbn/es';
expect(_calculateTopLevelDependency(plainDep)).toEqual('dep1');
expect(_calculateTopLevelDependency(foldedDep)).toEqual('@kbn/es');
expect(_calculateTopLevelDependency(otherFoldedDep)).toEqual('@kbn/es');
});
it('should exclude native modules', async () => {
readFile.mockImplementationOnce((path, options, cb) => {
cb(null, `require('fs')`);
});
const results = [];
await dependenciesParseStrategy(mockCwd, parseSingleFile, 'dep1/file.js', {}, results);
expect(results.length).toBe(0);
});
it('should return a dep from_modules', async () => {
readFile.mockImplementationOnce((path, options, cb) => {
cb(null, `require('dep_from_node_modules')`);
});
canRequire.mockImplementation((mockCwd, entry) => {
if (entry === `${mockCwd}dep1/dep_from_node_modules`) {
return false;
}
if (entry === 'dep_from_node_modules') {
return `${mockCwd}node_modules/dep_from_node_modules/index.js`;
}
});
const results = await dependenciesParseStrategy(mockCwd, parseSingleFile, 'dep1/file.js', {}, {});
expect(results[0]).toBe(`${mockCwd}node_modules/dep_from_node_modules/index.js`);
});
it('should return a relative dep file', async () => {
readFile.mockImplementationOnce((path, options, cb) => {
cb(null, `require('./relative_dep')`);
});
canRequire.mockImplementation((mockCwd, entry) => {
if (entry === `${mockCwd}dep1/relative_dep`) {
return `${entry}/index.js`;
}
return false;
});
const results = await dependenciesParseStrategy(mockCwd, parseSingleFile, 'dep1/file.js', {}, {});
expect(results[0]).toBe(`${mockCwd}dep1/relative_dep/index.js`);
});
afterAll(() => {
jest.clearAllMocks();
});
});

142
packages/kbn-babel-code-parser/src/visitors.js Normal file
View file

@ -0,0 +1,142 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import { matches } from 'lodash';
/**
* @notice
*
* This product has relied on ASTExplorer that is licensed under MIT.
*/
export function dependenciesVisitorsGenerator(dependenciesAcc) {
return (() => {
// This was built with help on an ast explorer and some ESTree docs
// like the babel parser ast spec and the main docs for the Esprima
// which is a complete and useful docs for the ESTree spec.
//
// https://astexplorer.net
// https://github.com/babel/babel/blob/master/packages/babel-parser/ast/spec.md
// https://esprima.readthedocs.io/en/latest/syntax-tree-format.html
// https://github.com/estree/estree
return {
// Visitors to traverse and found dependencies
// raw values on require + require.resolve
CallExpression: ({ node }) => {
// AST check for require expressions
const isRequire = (node) => {
return matches({
callee: {
type: 'Identifier',
name: 'require'
}
})(node);
};
// AST check for require.resolve expressions
const isRequireResolve = (node) => {
return matches({
callee: {
type: 'MemberExpression',
object: {
type: 'Identifier',
name: 'require'
},
property: {
type: 'Identifier',
name: 'resolve'
}
}
})(node);
};
// Get string values inside the expressions
// whether they are require or require.resolve
if (isRequire(node) || isRequireResolve(node)) {
const nodeArguments = node.arguments;
const reqArg = Array.isArray(nodeArguments) ? nodeArguments.shift() : null;
if (!reqArg) {
return;
}
if (reqArg.type === 'StringLiteral') {
dependenciesAcc.push(reqArg.value);
}
}
},
// Visitors to traverse and found dependencies
// raw values on import
ImportDeclaration: ({ node }) => {
// AST check for supported import expressions
const isImport = (node) => {
return matches({
type: 'ImportDeclaration',
source: {
type: 'StringLiteral'
}
})(node);
};
// Get string values from import expressions
if (isImport(node)) {
const importSource = node.source;
dependenciesAcc.push(importSource.value);
}
},
// Visitors to traverse and found dependencies
// raw values on export from
ExportNamedDeclaration: ({ node }) => {
// AST check for supported export from expressions
const isExportFrom = (node) => {
return matches({
type: 'ExportNamedDeclaration',
source: {
type: 'StringLiteral'
}
})(node);
};
// Get string values from export from expressions
if (isExportFrom(node)) {
const exportFromSource = node.source;
dependenciesAcc.push(exportFromSource.value);
}
},
// Visitors to traverse and found dependencies
// raw values on export * from
ExportAllDeclaration: ({ node }) => {
// AST check for supported export * from expressions
const isExportAllFrom = (node) => {
return matches({
type: 'ExportAllDeclaration',
source: {
type: 'StringLiteral'
}
})(node);
};
// Get string values from export * from expressions
if (isExportAllFrom(node)) {
const exportAllFromSource = node.source;
dependenciesAcc.push(exportAllFromSource.value);
}
}
};
})();
}

68
packages/kbn-babel-code-parser/src/visitors.test.js Normal file
View file

@ -0,0 +1,68 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import * as parser from '@babel/parser';
import traverse from '@babel/traverse';
import { dependenciesVisitorsGenerator } from './visitors';
const visitorsApplier = (code) => {
const result = [];
traverse(
parser.parse(code, {
sourceType: 'unambiguous',
plugins: ['exportDefaultFrom']
}),
dependenciesVisitorsGenerator(result)
);
return result;
};
describe('Code Parser Visitors', () => {
it('should get values from require', () => {
const rawCode = `/*foo*/require('dep1'); const bar = 1;`;
const foundDeps = visitorsApplier(rawCode);
expect(foundDeps[0] === 'dep1');
});
it('should get values from require.resolve', () => {
const rawCode = `/*foo*/require.resolve('dep2'); const bar = 1;`;
const foundDeps = visitorsApplier(rawCode);
expect(foundDeps[0] === 'dep2');
});
it('should get values from import', () => {
const rawCode = `/*foo*/import dep1 from 'dep1'; import dep2 from 'dep2';const bar = 1;`;
const foundDeps = visitorsApplier(rawCode);
expect(foundDeps[0] === 'dep1');
expect(foundDeps[1] === 'dep2');
});
it('should get values from export from', () => {
const rawCode = `/*foo*/export dep1 from 'dep1'; import dep2 from 'dep2';const bar = 1;`;
const foundDeps = visitorsApplier(rawCode);
expect(foundDeps[0] === 'dep1');
});
it('should get values from export * from', () => {
const rawCode = `/*foo*/export * from 'dep1'; export dep2 from 'dep2';const bar = 1;`;
const foundDeps = visitorsApplier(rawCode);
expect(foundDeps[0] === 'dep1');
expect(foundDeps[1] === 'dep2');
});
});

33
packages/kbn-babel-preset/common_babel_parser_options.js Normal file
View file

@ -0,0 +1,33 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
// The @babel/parser options documentation can be found here:
// https://babeljs.io/docs/en/babel-parser#options
module.exports = {
sourceType: 'unambiguous',
plugins: [
'asyncGenerators',
'classProperties',
'dynamicImport',
'exportDefaultFrom',
'exportNamespaceFrom',
'objectRestSpread',
'throwExpressions'
],
};

2
packages/kbn-babel-preset/package.json
View file

@ -9,7 +9,7 @@
"babel-plugin-transform-class-properties": "^6.24.1",
"babel-plugin-transform-define": "^1.3.0",
"babel-plugin-transform-object-rest-spread": "^6.26.0",
"babel-preset-env": "1.4.0",
"babel-preset-env": "^1.7.0",
"babel-preset-react": "^6.24.1"
}
}

4
packages/kbn-eslint-import-resolver-kibana/package.json
View file

@ -14,8 +14,8 @@
"eslint-import-resolver-node": "^0.3.2",
"eslint-import-resolver-webpack": "^0.10.1",
"glob-all": "^3.1.0",
"lru-cache": "^4.1.2",
"lru-cache": "^4.1.3",
"resolve": "^1.7.1",
"webpack": "3.6.0"
"webpack": "^4.23.1"
}
}

13
packages/kbn-interpreter/package.json
View file

@ -19,18 +19,19 @@
"@kbn/babel-preset": "1.0.0",
"@kbn/dev-utils": "1.0.0",
"babel-cli": "^6.26.0",
"babel-core": "^6.26.0",
"babel-loader": "7.1.2",
"babel-core": "6.26.3",
"babel-loader": "7.1.5",
"babel-plugin-transform-runtime": "^6.23.0",
"babel-polyfill": "6.20.0",
"css-loader": "0.28.7",
"css-loader": "1.0.0",
"del": "^3.0.0",
"getopts": "^2.2.3",
"pegjs": "0.9.0",
"sass-loader": "^7.1.0",
"style-loader": "0.19.0",
"style-loader": "0.23.1",
"supports-color": "^5.5.0",
"url-loader": "0.5.9",
"webpack": "3.6.0"
"url-loader": "1.1.2",
"webpack": "4.23.1",
"webpack-cli": "^3.1.2"
}
}

15
packages/kbn-interpreter/tasks/build/webpack.config.js
View file

@ -28,6 +28,7 @@ module.exports = function ({ sourceMaps }, { watch }) {
return {
devtool: sourceMaps ? 'inline-cheap-module-source-map' : undefined,
mode: 'none',
entry: {
'types/all': resolve(PLUGIN_SOURCE_DIR, 'types/register.js'),
'functions/common/all': resolve(PLUGIN_SOURCE_DIR, 'functions/common/register.js'),
@ -42,6 +43,16 @@ module.exports = function ({ sourceMaps }, { watch }) {
path: PLUGIN_BUILD_DIR,
filename: '[name].js', // Need long paths here.
libraryTarget: 'umd',
// Note: this is needed due to a not yet resolved bug on
// webpack 4 with umd modules generation.
// For now we have 2 quick workarounds: one is what is implemented
// below another is to change the libraryTarget to commonjs
//
// The issues can be followed on:
// https://github.com/webpack/webpack/issues/6642
// https://github.com/webpack/webpack/issues/6525
// https://github.com/webpack/webpack/issues/6677
globalObject: `(typeof self !== 'undefined' ? self : this)`,
},
resolve: {
@ -84,7 +95,7 @@ module.exports = function ({ sourceMaps }, { watch }) {
stats: 'errors-only',
plugins: [
function loaderFailHandler() {
function LoaderFailHandlerPlugin() {
if (!watch) {
return;
}
@ -93,7 +104,7 @@ module.exports = function ({ sourceMaps }, { watch }) {
// bails on error, including loader errors
// see https://github.com/webpack/webpack/issues/708, which does not fix loader errors
this.plugin('done', function (stats) {
this.hooks.done.tapPromise('LoaderFailHandlerPlugin', async stats => {
if (stats.hasErrors() || stats.hasWarnings()) {
lastBuildFailed = true;
return;

2
packages/kbn-plugin-helpers/package.json
View file

@ -18,7 +18,7 @@
"gulp-zip": "^4.1.0",
"inquirer": "^1.2.2",
"minimatch": "^3.0.4",
"node-sass": "^4.9.0",
"node-sass": "^4.9.4",
"through2": "^2.0.3",
"through2-map": "^3.0.0",
"vinyl-fs": "^3.0.0"

21286
packages/kbn-pm/dist/index.js vendored
View file

File diff suppressed because one or more lines are too long

15
packages/kbn-pm/package.json
View file

@ -25,7 +25,7 @@
"@types/log-symbols": "^2.0.0",
"@types/mkdirp": "^0.5.2",
"@types/ncp": "^2.0.1",
"@types/node": "^8.9.4",
"@types/node": "^8.10.38",
"@types/ora": "^1.3.2",
"@types/read-pkg": "^3.0.0",
"@types/strip-ansi": "^3.0.0",
@ -33,9 +33,9 @@
"@types/tempy": "^0.1.0",
"@types/wrap-ansi": "^2.0.14",
"@types/write-pkg": "^3.1.0",
"babel-core": "^6.26.0",
"babel-loader": "^7.1.2",
"babel-preset-env": "^1.6.1",
"babel-core": "^6.26.3",
"babel-loader": "^7.1.5",
"babel-preset-env": "^1.7.0",
"babel-preset-stage-3": "^6.24.1",
"chalk": "^2.4.1",
"cmd-shim": "^2.0.2",
@ -57,13 +57,14 @@
"read-pkg": "^3.0.0",
"rxjs": "^6.2.1",
"spawn-sync": "^1.0.15",
"string-replace-loader": "^1.3.0",
"string-replace-loader": "^2.1.1",
"strip-ansi": "^4.0.0",
"strong-log-transformer": "^2.0.0",
"tempy": "^0.2.1",
"ts-loader": "^3.5.0",
"ts-loader": "^5.2.2",
"typescript": "^3.0.3",
"webpack": "^3.11.0",
"webpack": "^4.23.1",
"webpack-cli": "^3.1.2",
"wrap-ansi": "^3.0.1",
"write-pkg": "^3.1.0"
},

11
packages/kbn-pm/src/config.ts
View file

@ -33,6 +33,17 @@ export function getProjectPaths(rootPath: string, options: IProjectPathOptions)
const projectPaths = [rootPath, resolve(rootPath, 'packages/*')];
// This is needed in order to install the dependencies for the declared
// plugin functional used in the selenium functional tests.
// As we are now using the webpack dll for the client vendors dependencies
// when we run the plugin functional tests against the distributable
// dependencies used by such plugins like @eui, react and react-dom can't
// be loaded from the dll as the context is different from the one declared
// into the webpack dll reference plugin.
// In anyway, have a plugin declaring their own dependencies is the
// correct and the expect behavior.
projectPaths.push(resolve(rootPath, 'test/plugin_functional/plugins/*'));
if (!ossOnly) {
projectPaths.push(resolve(rootPath, 'x-pack'));
projectPaths.push(resolve(rootPath, 'x-pack/plugins/*'));

5
packages/kbn-pm/webpack.config.js
View file

@ -20,6 +20,7 @@
const path = require('path');
module.exports = {
mode: 'production',
entry: {
index: './src/index.ts',
},
@ -83,4 +84,8 @@ module.exports = {
watchOptions: {
ignored: [/node_modules/, /vendor/],
},
optimization: {
minimize: false,
},
};

32
packages/kbn-ui-framework/package.json
View file

@ -33,30 +33,30 @@
"@elastic/eui": "0.0.23",
"@kbn/babel-preset": "1.0.0",
"autoprefixer": "6.5.4",
"babel-core": "6.21.0",
"babel-loader": "7.1.2",
"babel-polyfill": "6.20.0",
"babel-core": "^6.26.3",
"babel-loader": "^7.1.5",
"babel-polyfill": "^6.26.0",
"brace": "0.10.0",
"chalk": "^2.4.1",
"chokidar": "1.6.0",
"css-loader": "0.28.7",
"expose-loader": "0.7.3",
"file-loader": "1.1.4",
"css-loader": "^1.0.0",
"expose-loader": "^0.7.5",
"file-loader": "^2.0.0",
"grunt": "1.0.1",
"grunt-babel": "^7.0.0",
"grunt-contrib-clean": "^1.1.0",
"grunt-contrib-copy": "^1.0.0",
"highlight.js": "9.0.0",
"html": "1.0.0",
"html-loader": "0.5.1",
"imports-loader": "0.7.1",
"html-loader": "^0.5.5",
"imports-loader": "^0.8.0",
"jquery": "^3.3.1",
"keymirror": "0.1.1",
"moment": "^2.20.1",
"node-sass": "4.5.3",
"postcss": "^5.0.10",
"postcss-loader": "2.0.6",
"raw-loader": "0.5.1",
"node-sass": "^4.9.4",
"postcss": "^7.0.5",
"postcss-loader": "^3.0.0",
"raw-loader": "^0.5.1",
"react-dom": "^16.2.0",
"react-redux": "^5.0.6",
"react-router": "^3.2.0",
@ -64,11 +64,11 @@
"react-router-redux": "^4.0.8",
"redux": "3.7.2",
"redux-thunk": "2.2.0",
"sass-loader": "6.0.6",
"sass-loader": "^7.1.0",
"sinon": "^5.0.7",
"style-loader": "0.19.0",
"webpack": "3.6.0",
"webpack-dev-server": "2.9.1",
"style-loader": "^0.23.1",
"webpack": "^4.23.1",
"webpack-dev-server": "^3.1.10",
"yeoman-generator": "1.1.1",
"yo": "2.0.3"
}

11
src/dev/build/README.md
View file

@ -39,3 +39,14 @@ The majority of this logic is extracted from the grunt build that has existed fo
[lib/build.js]: ./lib/build.js
[build_distributables.js]: ./build_distributables.js
[../tooling_log/tooling_log.js]: ../tooling_log/tooling_log.js
# Client Node Modules Cleaning
We have introduced in our bundle a webpack dll for the client vendor modules in order to improve
the optimization time both in dev and in production. As for those modules we already have the
code into the vendors.bundle.dll.js we have decided to delete those bundled modules from the
distributable node_modules folder. However, in order to accomplish this, we need to exclude
every node_module used in the server side code. This logic is performed
under `nodejs_modules/clean_client_modules_on_dll_task.js`. In case we need to add any new cli
or any other piece of server code other than `x-pack` or `core_plugins` we'll need
to update the globs present on `clean_client_modules_on_dll_task.js` accordingly.

6
src/dev/build/build_distributables.js
View file

@ -21,6 +21,8 @@ import { getConfig, createRunner } from './lib';
import {
BuildPackagesTask,
CleanClientModulesOnDLLTask,
CleanEmptyFoldersTask,
CleanExtraBinScriptsTask,
CleanExtraBrowsersTask,
CleanExtraFilesFromModulesTask,
@ -116,8 +118,10 @@ export async function buildDistributables(options) {
await run(UpdateLicenseFileTask);
await run(RemovePackageJsonDepsTask);
await run(TranspileScssTask);
await run(CleanExtraFilesFromModulesTask);
await run(OptimizeBuildTask);
await run(CleanClientModulesOnDLLTask);
await run(CleanExtraFilesFromModulesTask);
await run(CleanEmptyFoldersTask);
/**
* copy generic build outputs into platform-specific build

2
src/dev/build/lib/exec.js
View file

@ -26,6 +26,7 @@ export async function exec(log, cmd, args, options = {}) {
const {
level = 'debug',
cwd,
env,
exitAfter,
} = options;
@ -34,6 +35,7 @@ export async function exec(log, cmd, args, options = {}) {
const proc = execa(cmd, args, {
stdio: ['ignore', 'pipe', 'pipe'],
cwd,
env,
});
await watchStdioForLine(proc, line => log[level](line), exitAfter);

23
src/dev/build/lib/fs.js
View file

@ -27,6 +27,7 @@ import vfs from 'vinyl-fs';
import { promisify } from 'bluebird';
import mkdirpCb from 'mkdirp';
import del from 'del';
import deleteEmpty from 'delete-empty';
import { createPromiseFromStreams, createMapStream } from '../../../utils';
import { Extract } from 'tar';
@ -112,6 +113,28 @@ export async function deleteAll(log, patterns) {
log.verbose('Deleted:', longInspect(files));
}
export async function deleteEmptyFolders(log, rootFolderPath, foldersToKeep) {
if (typeof rootFolderPath !== 'string') {
throw new TypeError('Expected root folder to be a string path');
}
log.debug('Deleting all empty folders and their children recursively starting on ', rootFolderPath);
assertAbsolute(rootFolderPath.startsWith('!') ? rootFolderPath.slice(1) : rootFolderPath);
// Delete empty is used to gather all the empty folders and
// then we use del to actually delete them
const emptyFoldersList = await deleteEmpty(rootFolderPath, { dryRun: true });
const foldersToDelete = emptyFoldersList.filter((folderToDelete) => {
return !foldersToKeep.some(folderToKeep => folderToDelete.includes(folderToKeep));
});
const deletedEmptyFolders = await del(foldersToDelete, {
concurrency: 4
});
log.debug('Deleted %d empty folders', deletedEmptyFolders.length);
log.verbose('Deleted:', longInspect(deletedEmptyFolders));
}
export async function copyAll(sourceDir, destination, options = {}) {
const {
select = ['**/*'],

1
src/dev/build/lib/index.js
View file

@ -30,6 +30,7 @@ export {
getFileHash,
untar,
deleteAll,
deleteEmptyFolders,
} from './fs';
export { scanDelete } from './scan_delete';
export { scanCopy } from './scan_copy';

20
src/dev/build/tasks/clean_tasks.js
View file

@ -19,7 +19,7 @@
import minimatch from 'minimatch';
import { deleteAll, scanDelete } from '../lib';
import { deleteAll, deleteEmptyFolders, scanDelete } from '../lib';
export const CleanTask = {
global: true,
@ -233,3 +233,21 @@ export const CleanExtraBrowsersTask = {
}
},
};
export const CleanEmptyFoldersTask = {
description: 'Cleaning all empty folders recursively',
async run(config, log, build) {
// Delete every single empty folder from
// the distributable except the plugins
// and data folder.
await deleteEmptyFolders(
log,
build.resolvePath('.'),
[
build.resolvePath('plugins'),
build.resolvePath('data')
]
);
},
};

1
src/dev/build/tasks/index.js
View file

@ -28,6 +28,7 @@ export * from './create_readme_task';
export * from './install_dependencies_task';
export * from './license_file_task';
export * from './nodejs';
export * from './nodejs_modules';
export * from './notice_file_task';
export * from './optimize_task';
export * from './os_packages';

83
src/dev/build/tasks/nodejs_modules/clean_client_modules_on_dll_task.js Normal file
View file

@ -0,0 +1,83 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import { getDllEntries, cleanDllModuleFromEntryPath, writeEmptyFileForDllEntry } from './webpack_dll';
import { getDependencies } from './get_dependencies';
import globby from 'globby';
import normalizePosixPath from 'normalize-path';
export const CleanClientModulesOnDLLTask = {
description:
'Cleaning client node_modules bundled into the DLL',
async run(config, log, build) {
const baseDir = normalizePosixPath(build.resolvePath('.'));
const kbnPkg = config.getKibanaPkg();
const kbnPkgDependencies = (kbnPkg && kbnPkg.dependencies) || {};
const kbnWebpackLoaders = Object.keys(kbnPkgDependencies).filter(dep => !!dep.includes('-loader'));
// Define the entry points for the server code in order to
// start here later looking for the server side dependencies
const mainCodeEntries = [
`${baseDir}/src/cli`,
`${baseDir}/src/cli_keystore`,
`${baseDir}/src/cli_plugin`,
`${baseDir}/node_modules/x-pack`,
...kbnWebpackLoaders.map(loader => `${baseDir}/node_modules/${loader}`)
];
const discoveredLegacyCorePluginEntries = await globby([
`${baseDir}/src/legacy/core_plugins/*/index.js`,
`!${baseDir}/src/legacy/core_plugins/**/public`
]);
const discoveredPluginEntries = await globby([
`${baseDir}/src/plugins/*/index.js`,
`!${baseDir}/src/plugins/**/public`
]);
// Compose all the needed entries
const serverEntries = [ ...mainCodeEntries, ...discoveredLegacyCorePluginEntries, ...discoveredPluginEntries];
// Get the dependencies found searching through the server
// side code entries that were provided
const serverDependencies = await getDependencies(baseDir, serverEntries);
// Consider this as our whiteList for the modules we can't delete
const whiteListedModules = [
...serverDependencies,
...kbnWebpackLoaders
];
// Resolve the client vendors dll manifest path
const dllManifestPath = `${baseDir}/dlls/vendors.manifest.dll.json`;
// Get dll entries filtering out the ones
// from any whitelisted module
const dllEntries = await getDllEntries(dllManifestPath, whiteListedModules);
for (const relativeEntryPath of dllEntries) {
const entryPath = `${baseDir}/${relativeEntryPath}`;
// Clean a module included into the dll
// and then write a blank file for each
// entry file present into the dll
await cleanDllModuleFromEntryPath(log, entryPath);
await writeEmptyFileForDllEntry(entryPath);
}
}
};

27
src/dev/build/tasks/nodejs_modules/get_dependencies.js Normal file
View file

@ -0,0 +1,27 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import { parseEntries, dependenciesParseStrategy } from '@kbn/babel-code-parser';
export async function getDependencies(cwd, entries) {
// Return the dependencies retrieve from the
// provided code entries (sanitized) and
// parseStrategy (dependencies one)
return Object.keys(await parseEntries(cwd, entries, dependenciesParseStrategy, {}));
}

20
src/dev/build/tasks/nodejs_modules/index.js Normal file
View file

@ -0,0 +1,20 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
export { CleanClientModulesOnDLLTask } from './clean_client_modules_on_dll_task';

103
src/dev/build/tasks/nodejs_modules/webpack_dll.js Normal file
View file

@ -0,0 +1,103 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import { deleteAll, read, write } from '../../lib';
import { dirname, sep } from 'path';
import pkgUp from 'pkg-up';
import globby from 'globby';
export async function getDllEntries(manifestPath, whiteListedModules) {
const manifest = JSON.parse(await read(manifestPath));
if (!manifest || !manifest.content) {
// It should fails because if we don't have the manifest file
// or it is malformed something wrong is happening and we
// should stop
throw new Error(`The following dll manifest doesn't exists: ${manifestPath}`);
}
const modules = Object.keys(manifest.content);
if (!modules.length) {
// It should fails because if we don't have any
// module inside the client vendors dll something
// wrong is happening and we should stop too
throw new Error(`The following dll manifest is reporting an empty dll: ${manifestPath}`);
}
// Only includes modules who are not in the white list of modules
// and that are node_modules
return modules.filter(entry => {
const isWhiteListed = whiteListedModules.some(nonEntry => entry.includes(`node_modules${sep}${nonEntry}${sep}`));
const isNodeModule = entry.includes('node_modules');
return !isWhiteListed && isNodeModule;
});
}
export async function cleanDllModuleFromEntryPath(logger, entryPath) {
const modulePkgPath = await pkgUp(entryPath);
const modulePkg = JSON.parse(await read(modulePkgPath));
const moduleDir = dirname(modulePkgPath);
// Cancel the cleanup for this module as it
// was already done.
if (modulePkg.cleaned) {
return;
}
// Clear dependencies from dll module package.json
if (modulePkg.dependencies) {
modulePkg.dependencies = {};
}
// Clear devDependencies from dll module package.json
if (modulePkg.devDependencies) {
modulePkg.devDependencies = {};
}
// Delete module contents. It will delete everything
// excepts package.json, images and css
//
// NOTE: We can't use cwd option with globby
// until the following issue gets closed
// https://github.com/sindresorhus/globby/issues/87
const deletePatterns = await globby([
`${moduleDir}/**`,
`!${moduleDir}/**/*.+(css)`,
`!${moduleDir}/**/*.+(gif|ico|jpeg|jpg|tiff|tif|svg|png|webp)`,
`!${modulePkgPath}`,
]);
await deleteAll(logger, deletePatterns);
// Mark this module as cleaned
modulePkg.cleaned = true;
// Rewrite modified package.json
await write(
modulePkgPath,
JSON.stringify(modulePkg, null, 2)
);
}
export async function writeEmptyFileForDllEntry(entryPath) {
await write(
entryPath,
''
);
}

98
src/dev/build/tasks/nodejs_modules/webpack_dll.test.js Normal file
View file

@ -0,0 +1,98 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import { read } from '../../lib';
import { getDllEntries } from './webpack_dll';
jest.mock('../../lib', () => ({
read: jest.fn()
}));
const manifestContentMock = JSON.stringify({
'name': 'vendors',
'content': {
'/mock/node_modules/dep1': {},
'/mock/node_modules/dep2': {},
'/mock/node_modules/dep3': {},
'/mock/tmp/dep2': {}
}
});
const emptyManifestContentMock = JSON.stringify({
'name': 'vendors',
'content': {}
});
const noManifestMock = JSON.stringify(null);
const noContentFieldManifestMock = JSON.stringify({
'name': 'vendors',
});
describe('Webpack DLL Build Tasks Utils', () => {
it('should get dll entries correctly', async () => {
read.mockImplementationOnce(async () => manifestContentMock);
const mockManifestPath = '/mock/mock_dll_manifest.json';
const mockModulesWhitelist = [ 'dep1' ];
const dllEntries = await getDllEntries(mockManifestPath, mockModulesWhitelist);
expect(dllEntries).toEqual(expect.arrayContaining(['/mock/node_modules/dep2', '/mock/node_modules/dep3']));
});
it('should throw an error for no manifest file', async () => {
read.mockImplementationOnce(async () => noManifestMock);
const mockManifestPath = '/mock/mock_dll_manifest.json';
try {
await getDllEntries(mockManifestPath, []);
} catch (error) {
expect(error.message).toEqual(`The following dll manifest doesn't exists: /mock/mock_dll_manifest.json`);
}
});
it('should throw an error for no manifest content field', async () => {
read.mockImplementation(async () => noContentFieldManifestMock);
const mockManifestPath = '/mock/mock_dll_manifest.json';
try {
await getDllEntries(mockManifestPath, []);
} catch (error) {
expect(error.message).toEqual(`The following dll manifest doesn't exists: /mock/mock_dll_manifest.json`);
}
});
it('should throw an error for manifest file without any content', async () => {
read.mockImplementation(async () => emptyManifestContentMock);
const mockManifestPath = '/mock/mock_dll_manifest.json';
try {
await getDllEntries(mockManifestPath, []);
} catch (error) {
expect(error.message).toEqual(`The following dll manifest is reporting an empty dll: /mock/mock_dll_manifest.json`);
}
});
afterAll(() => {
jest.clearAllMocks();
});
});

3
src/dev/build/tasks/optimize_task.js
View file

@ -46,6 +46,9 @@ export const OptimizeBuildTask = {
await exec(log, kibanaScript, kibanaArgs, {
cwd: build.resolvePath('.'),
env: {
FORCE_DLL_CREATION: 'true'
},
});
// clean up temporary node install

2
src/dev/notice/generate_notice_from_source.js
View file

@ -42,7 +42,7 @@ export async function generateNoticeFromSource({ productName, directory, log })
cwd: directory,
nodir: true,
ignore: [
'{node_modules,build,target,dist,optimize}/**',
'{node_modules,build,target,dist,optimize,dlls}/**',
'packages/*/{node_modules,build,target,dist}/**',
'x-pack/{node_modules,build,target,dist,optimize}/**',
'x-pack/packages/*/{node_modules,build,target,dist}/**',

286
src/optimize/base_optimizer.js
View file

@ -18,13 +18,14 @@
*/
import { writeFile } from 'fs';
import os from 'os';
import Boom from 'boom';
import ExtractTextPlugin from 'extract-text-webpack-plugin';
import MiniCssExtractPlugin from 'mini-css-extract-plugin';
import UglifyJsPlugin from 'uglifyjs-webpack-plugin';
import webpack from 'webpack';
import Stats from 'webpack/lib/Stats';
import webpackMerge from 'webpack-merge';
import { DynamicDllPlugin } from './dynamic_dll_plugin';
import { defaults } from 'lodash';
@ -37,9 +38,14 @@ const BABEL_PRESET_PATH = require.resolve('@kbn/babel-preset/webpack_preset');
const BABEL_EXCLUDE_RE = [
/[\/\\](webpackShims|node_modules|bower_components)[\/\\]/,
];
const STATS_WARNINGS_FILTER = new RegExp([
'(export .* was not found in)',
'|(chunk .* \\[mini-css-extract-plugin\\]\\\nConflicting order between:)'
].join(''));
export default class BaseOptimizer {
constructor(opts) {
this.log = opts.log || (() => null);
this.uiBundles = opts.uiBundles;
this.profile = opts.profile || false;
@ -57,22 +63,36 @@ export default class BaseOptimizer {
break;
}
this.unsafeCache = opts.unsafeCache || false;
if (typeof this.unsafeCache === 'string') {
this.unsafeCache = [
new RegExp(this.unsafeCache.slice(1, -1))
];
}
}
async initCompiler() {
if (this.compiler) return this.compiler;
async init() {
if (this.compiler) {
return this;
}
const compilerConfig = this.getConfig();
this.compiler = webpack(compilerConfig);
this.compiler.plugin('done', stats => {
if (!this.profile) return;
// register the webpack compiler hooks
// for the base optimizer
this.registerCompilerHooks();
return this;
}
registerCompilerHooks() {
this.registerCompilerDoneHook();
}
registerCompilerDoneHook() {
this.compiler.hooks.done.tap('base_optimizer-done', stats => {
// We are not done while we have an additional
// compilation pass to run
// We also don't need to emit the stats if we don't have
// the profile option set
if (!this.profile || stats.compilation.needAdditionalPass) {
return;
}
const path = this.uiBundles.resolvePath('stats.json');
const content = JSON.stringify(stats.toJson());
@ -80,41 +100,38 @@ export default class BaseOptimizer {
if (err) throw err;
});
});
return this.compiler;
}
getConfig() {
function getStyleLoaders(preProcessors = [], postProcessors = []) {
return ExtractTextPlugin.extract({
fallback: {
loader: 'style-loader'
},
use: [
...postProcessors,
{
loader: 'css-loader',
options: {
// importLoaders needs to know the number of loaders that follow this one,
// so we add 1 (for the postcss-loader) to the length of the preProcessors
// array that we merge into this array
importLoaders: 1 + preProcessors.length,
},
},
{
loader: 'postcss-loader',
options: {
config: {
path: POSTCSS_CONFIG_PATH,
},
},
},
...preProcessors,
],
});
function getStyleLoaderExtractor() {
return [
MiniCssExtractPlugin.loader
];
}
const nodeModulesPath = fromRoot('node_modules');
function getStyleLoaders(preProcessors = [], postProcessors = []) {
return [
...postProcessors,
{
loader: 'css-loader',
options: {
// importLoaders needs to know the number of loaders that follow this one,
// so we add 1 (for the postcss-loader) to the length of the preProcessors
// array that we merge into this array
importLoaders: 1 + preProcessors.length,
},
},
{
loader: 'postcss-loader',
options: {
config: {
path: POSTCSS_CONFIG_PATH,
},
},
},
...preProcessors,
];
}
/**
* Adds a cache loader if we're running in dev mode. The reason we're not adding
@ -157,8 +174,11 @@ export default class BaseOptimizer {
};
const commonConfig = {
mode: 'development',
node: { fs: 'empty' },
context: fromRoot('.'),
parallelism: os.cpus().length - 1,
cache: true,
entry: this.uiBundles.toWebpackEntries(),
devtool: this.sourceMaps,
@ -172,26 +192,30 @@ export default class BaseOptimizer {
devtoolModuleFilenameTemplate: '[absolute-resource-path]'
},
optimization: {
splitChunks: {
cacheGroups: {
commons: {
name: 'commons',
chunks: 'initial',
minChunks: 2,
reuseExistingChunk: true
}
}
},
noEmitOnErrors: true
},
plugins: [
new ExtractTextPlugin('[name].style.css', {
allChunks: true
new DynamicDllPlugin({
uiBundles: this.uiBundles,
log: this.log
}),
new webpack.optimize.CommonsChunkPlugin({
name: 'commons',
filename: 'commons.bundle.js',
minChunks: 2,
new MiniCssExtractPlugin({
filename: '[name].style.css',
}),
new webpack.optimize.CommonsChunkPlugin({
name: 'vendors',
filename: 'vendors.bundle.js',
// only combine node_modules from Kibana
minChunks: module => module.context && module.context.indexOf(nodeModulesPath) !== -1
}),
new webpack.NoEmitOnErrorsPlugin(),
// replace imports for `uiExports/*` modules with a synthetic module
// created by create_ui_exports_module.js
new webpack.NormalModuleReplacementPlugin(/^uiExports\//, (resource) => {
@ -224,14 +248,17 @@ export default class BaseOptimizer {
rules: [
{
test: /\.less$/,
use: getStyleLoaders(
['less-loader'],
maybeAddCacheLoader('less', [])
),
use: [
...getStyleLoaderExtractor(),
...getStyleLoaders(['less-loader'], maybeAddCacheLoader('less', []))
],
},
{
test: /\.css$/,
use: getStyleLoaders(),
use: [
...getStyleLoaderExtractor(),
...getStyleLoaders([], maybeAddCacheLoader('css', []))
],
},
{
test: /\.(html|tmpl)$/,
@ -277,9 +304,15 @@ export default class BaseOptimizer {
'node_modules',
fromRoot('node_modules'),
],
alias: this.uiBundles.getAliases(),
unsafeCache: this.unsafeCache,
alias: this.uiBundles.getAliases()
},
performance: {
// NOTE: we are disabling this as those hints
// are more tailored for the final bundles result
// and not for the webpack compilations performance itself
hints: false
}
};
// when running from the distributable define an environment variable we can use
@ -324,16 +357,6 @@ export default class BaseOptimizer {
]
},
stats: {
// when typescript doesn't do a full type check, as we have the ts-loader
// configured here, it does not have enough information to determine
// whether an imported name is a type or not, so when the name is then
// exported, typescript has no choice but to emit the export. Fortunately,
// the extraneous export should not be harmful, so we just suppress these warnings
// https://github.com/TypeStrong/ts-loader#transpileonly-boolean-defaultfalse
warningsFilter: /export .* was not found in/
},
resolve: {
extensions: ['.ts', '.tsx'],
},
@ -366,48 +389,49 @@ export default class BaseOptimizer {
// in production we set the process.env.NODE_ENV and uglify our bundles
const productionConfig = {
plugins: [
new webpack.DefinePlugin({
'process.env': {
'NODE_ENV': '"production"'
}
}),
new UglifyJsPlugin({
parallel: true,
sourceMap: false,
uglifyOptions: {
compress: {
// the following is required for dead-code the removal
// check in React DevTools
mode: 'production',
optimization: {
minimize: true,
minimizer: [
new UglifyJsPlugin({
parallel: true,
sourceMap: false,
uglifyOptions: {
compress: {
// The following is required for dead-code the removal
// check in React DevTools
//
// default
unused: true,
dead_code: true,
conditionals: true,
evaluate: true,
unused: true,
dead_code: true,
conditionals: true,
evaluate: true,
comparisons: false,
sequences: false,
properties: false,
drop_debugger: false,
booleans: false,
loops: false,
toplevel: false,
top_retain: false,
hoist_funs: false,
if_return: false,
join_vars: false,
collapse_vars: false,
reduce_vars: false,
warnings: false,
negate_iife: false,
keep_fnames: true,
keep_infinity: true,
side_effects: false
},
mangle: false
}
}),
]
// changed
keep_fnames: true,
keep_infinity: true,
comparisons: false,
sequences: false,
properties: false,
drop_debugger: false,
booleans: false,
loops: false,
toplevel: false,
top_retain: false,
hoist_funs: false,
if_return: false,
join_vars: false,
collapse_vars: false,
reduce_vars: false,
warnings: false,
negate_iife: false,
side_effects: false
},
mangle: false
}
}),
]
}
};
return webpackMerge(
@ -421,15 +445,43 @@ export default class BaseOptimizer {
);
}
isFailure(stats) {
if (stats.hasErrors()) {
return true;
}
const { warnings } = stats.toJson({ all: false, warnings: true });
// 1 - when typescript doesn't do a full type check, as we have the ts-loader
// configured here, it does not have enough information to determine
// whether an imported name is a type or not, so when the name is then
// exported, typescript has no choice but to emit the export. Fortunately,
// the extraneous export should not be harmful, so we just suppress these warnings
// https://github.com/TypeStrong/ts-loader#transpileonly-boolean-defaultfalse
//
// 2 - Mini Css Extract plugin tracks the order for each css import we have
// through the project (and it's successive imports) since version 0.4.2.
// In case we have the same imports more than one time with different
// sequences, this plugin will throw a warning. This should not be harmful,
// but the an issue was opened and can be followed on:
// https://github.com/webpack-contrib/mini-css-extract-plugin/issues/250#issuecomment-415345126
const filteredWarnings = Stats.filterWarnings(warnings, STATS_WARNINGS_FILTER);
return filteredWarnings.length > 0;
}
failedStatsToError(stats) {
const details = stats.toString(defaults(
{ colors: true },
{ colors: true, warningsFilter: STATS_WARNINGS_FILTER },
Stats.presetToOptions('minimal')
));
return Boom.internal(
`Optimizations failure.\n${details.split('\n').join('\n ')}\n`,
stats.toJson(Stats.presetToOptions('detailed'))
stats.toJson(defaults({
warningsFilter: STATS_WARNINGS_FILTER,
...Stats.presetToOptions('detailed')
}))
);
}
}

44
src/optimize/bundles_route/__tests__/bundles_route.js
View file

@ -42,7 +42,8 @@ describe('optimizer/bundle route', () => {
function createServer(options = {}) {
const {
bundlesPath = outputFixture,
regularBundlesPath = outputFixture,
dllBundlesPath = outputFixture,
basePublicPath = ''
} = options;
@ -50,7 +51,8 @@ describe('optimizer/bundle route', () => {
server.register([Inert]);
server.route(createBundlesRoute({
bundlesPath,
regularBundlesPath,
dllBundlesPath,
basePublicPath,
}));
@ -60,28 +62,32 @@ describe('optimizer/bundle route', () => {
afterEach(() => sandbox.restore());
describe('validation', () => {
it('validates that bundlesPath is an absolute path', () => {
it('validates that regularBundlesPath is an absolute path', () => {
expect(() => {
createBundlesRoute({
bundlesPath: null,
regularBundlesPath: null,
dllBundlesPath: '/absolute/path',
basePublicPath: ''
});
}).to.throwError(/absolute path/);
expect(() => {
createBundlesRoute({
bundlesPath: './relative',
regularBundlesPath: './relative',
dllBundlesPath: '/absolute/path',
basePublicPath: ''
});
}).to.throwError(/absolute path/);
expect(() => {
createBundlesRoute({
bundlesPath: 1234,
regularBundlesPath: 1234,
dllBundlesPath: '/absolute/path',
basePublicPath: ''
});
}).to.throwError(/absolute path/);
expect(() => {
createBundlesRoute({
bundlesPath: '/absolute/path',
regularBundlesPath: '/absolute/path',
dllBundlesPath: '/absolute/path',
basePublicPath: ''
});
}).to.not.throwError();
@ -89,37 +95,43 @@ describe('optimizer/bundle route', () => {
it('validates that basePublicPath is valid', () => {
expect(() => {
createBundlesRoute({
bundlesPath: '/bundles',
regularBundlesPath: '/bundles',
dllBundlesPath: '/absolute/path',
basePublicPath: 123
});
}).to.throwError(/string/);
expect(() => {
createBundlesRoute({
bundlesPath: '/bundles',
regularBundlesPath: '/bundles',
dllBundlesPath: '/absolute/path',
basePublicPath: {}
});
}).to.throwError(/string/);
expect(() => {
createBundlesRoute({
bundlesPath: '/bundles',
regularBundlesPath: '/bundles',
dllBundlesPath: '/absolute/path',
basePublicPath: '/a/'
});
}).to.throwError(/start and not end with a \//);
expect(() => {
createBundlesRoute({
bundlesPath: '/bundles',
regularBundlesPath: '/bundles',
dllBundlesPath: '/absolute/path',
basePublicPath: 'a/'
});
}).to.throwError(/start and not end with a \//);
expect(() => {
createBundlesRoute({
bundlesPath: '/bundles',
regularBundlesPath: '/bundles',
dllBundlesPath: '/absolute/path',
basePublicPath: '/a'
});
}).to.not.throwError();
expect(() => {
createBundlesRoute({
bundlesPath: '/bundles',
regularBundlesPath: '/bundles',
dllBundlesPath: '/absolute/path',
basePublicPath: ''
});
}).to.not.throwError();
@ -215,7 +227,7 @@ describe('optimizer/bundle route', () => {
});
});
describe('js file outside bundlesPath', () => {
describe('js file outside regularBundlesPath', () => {
it('responds with a 404', async () => {
const server = createServer();
@ -249,10 +261,10 @@ describe('optimizer/bundle route', () => {
});
});
describe('missing bundlesPath', () => {
describe('missing regularBundlesPath', () => {
it('responds with 404', async () => {
const server = createServer({
bundlesPath: resolve(__dirname, 'fixtures/not_really_output')
regularBundlesPath: resolve(__dirname, 'fixtures/not_really_output')
});
const response = await server.inject({

34
src/optimize/bundles_route/bundles_route.js
View file

@ -18,29 +18,35 @@
*/
import { isAbsolute, extname } from 'path';
import LruCache from 'lru-cache';
import { createDynamicAssetResponse } from './dynamic_asset_response';
/**
* Creates a route that serves files from `bundlesPath`. If the
* Creates the routes that serves files from `bundlesPath` or from
* `dllBundlesPath` (if they are dll bundle's related files). If the
* file is js or css then it is searched for instances of
* PUBLIC_PATH_PLACEHOLDER and replaces them with `publicPath`.
*
* @param {Object} options
* @property {string} options.bundlesPath
* @property {string} options.regularBundlesPath
* @property {string} options.dllBundlesPath
* @property {string} options.basePublicPath
* @return {Hapi.RouteConfig}
*
* @return Array.of({Hapi.Route})
*/
export function createBundlesRoute({ bundlesPath, basePublicPath }) {
export function createBundlesRoute({ regularBundlesPath, dllBundlesPath, basePublicPath }) {
// rather than calculate the fileHash on every request, we
// provide a cache object to `createDynamicAssetResponse()` that
// will store the 100 most recently used hashes.
const fileHashCache = new LruCache(100);
if (typeof bundlesPath !== 'string' || !isAbsolute(bundlesPath)) {
throw new TypeError('bundlesPath must be an absolute path to the directory containing the bundles');
if (typeof regularBundlesPath !== 'string' || !isAbsolute(regularBundlesPath)) {
throw new TypeError('regularBundlesPath must be an absolute path to the directory containing the regular bundles');
}
if (typeof dllBundlesPath !== 'string' || !isAbsolute(dllBundlesPath)) {
throw new TypeError('dllBundlesPath must be an absolute path to the directory containing the dll bundles');
}
if (typeof basePublicPath !== 'string') {
@ -51,15 +57,23 @@ export function createBundlesRoute({ bundlesPath, basePublicPath }) {
throw new TypeError('basePublicPath must be empty OR start and not end with a /');
}
return [
buildRouteForBundles(basePublicPath, '/bundles/', regularBundlesPath, fileHashCache),
buildRouteForBundles(basePublicPath, '/dlls/', dllBundlesPath, fileHashCache),
];
}
function buildRouteForBundles(basePublicPath, routePath, bundlesPath, fileHashCache) {
return {
method: 'GET',
path: '/bundles/{path*}',
path: `${routePath}{path*}`,
config: {
auth: false,
ext: {
onPreHandler: {
method(request, h) {
const ext = extname(request.params.path);
if (ext !== '.js' && ext !== '.css') {
return h.continue;
}
@ -69,7 +83,7 @@ export function createBundlesRoute({ bundlesPath, basePublicPath }) {
h,
bundlesPath,
fileHashCache,
publicPath: `${basePublicPath}/bundles/`
publicPath: `${basePublicPath}${routePath}`
});
}
}

1
src/optimize/bundles_route/index.js
View file

@ -18,3 +18,4 @@
*/
export { createBundlesRoute } from './bundles_route';
export { createProxyBundlesRoute } from './proxy_bundles_route';

41
src/optimize/bundles_route/proxy_bundles_route.js Normal file
View file

@ -0,0 +1,41 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
export function createProxyBundlesRoute({ host, port }) {
return [
buildProxyRouteForBundles('/bundles/', host, port),
buildProxyRouteForBundles('/dlls/', host, port)
];
}
function buildProxyRouteForBundles(routePath, host, port) {
return {
path: `${routePath}{path*}`,
method: 'GET',
handler: {
proxy: {
host,
port,
passThrough: true,
xforward: true
}
},
config: { auth: false }
};
}

214
src/optimize/dynamic_dll_plugin/dll_compiler.js Normal file
View file

@ -0,0 +1,214 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import { configModel } from './dll_config_model';
import { fromRoot } from '../../utils';
import { PUBLIC_PATH_PLACEHOLDER } from '../public_path_placeholder';
import fs from 'fs';
import mkdirp from 'mkdirp';
import webpack from 'webpack';
import { promisify } from 'util';
import path from 'path';
const readFileAsync = promisify(fs.readFile);
const mkdirpAsync = promisify(mkdirp);
const existsAsync = promisify(fs.exists);
const writeFileAsync = promisify(fs.writeFile);
export class DllCompiler {
static getRawDllConfig(alias = {}, noParseRules = []) {
return {
alias,
noParseRules,
context: fromRoot('.'),
entryName: 'vendors',
dllName: '[name]',
manifestName: '[name]',
styleName: '[name]',
entryExt: '.entry.dll.js',
dllExt: '.bundle.dll.js',
manifestExt: '.manifest.dll.json',
styleExt: '.style.dll.css',
outputPath: fromRoot('./dlls'),
publicPath: PUBLIC_PATH_PLACEHOLDER
};
}
constructor(uiBundles, log) {
this.rawDllConfig = DllCompiler.getRawDllConfig(
uiBundles.getAliases(),
uiBundles.getWebpackNoParseRules()
);
this.log = log || (() => null);
}
async init() {
await this.ensureEntryFileExists();
await this.ensureManifestFileExists();
await this.ensureOutputPathExists();
}
async upsertEntryFile(content) {
await this.upsertFile(this.getEntryPath(), content);
}
async upsertFile(filePath, content = '') {
await this.ensurePathExists(filePath);
await writeFileAsync(filePath, content, 'utf8');
}
getDllPath() {
return this.resolvePath(
`${this.rawDllConfig.entryName}${this.rawDllConfig.dllExt}`
);
}
getEntryPath() {
return this.resolvePath(
`${this.rawDllConfig.entryName}${this.rawDllConfig.entryExt}`
);
}
getManifestPath() {
return this.resolvePath(
`${this.rawDllConfig.entryName}${this.rawDllConfig.manifestExt}`
);
}
getStylePath() {
return this.resolvePath(
`${this.rawDllConfig.entryName}${this.rawDllConfig.styleExt}`
);
}
async ensureEntryFileExists() {
await this.ensureFileExists(this.getEntryPath());
}
async ensureManifestFileExists() {
await this.ensureFileExists(
this.getManifestPath(),
JSON.stringify({
name: this.rawDllConfig.entryName,
content: {}
})
);
}
async ensureStyleFileExists() {
await this.ensureFileExists(this.getStylePath());
}
async ensureFileExists(filePath, content) {
const exists = await this.ensurePathExists(filePath);
if (!exists) {
await this.upsertFile(filePath, content);
}
}
async ensurePathExists(filePath) {
const exists = await existsAsync(filePath);
if (!exists) {
await mkdirpAsync(path.dirname(filePath));
}
return exists;
}
async ensureOutputPathExists() {
await this.ensurePathExists(this.rawDllConfig.outputPath);
}
dllExistsSync() {
return this.existsSync(this.getDllPath());
}
existsSync(filePath) {
return fs.existsSync(filePath);
}
resolvePath() {
return path.resolve(this.rawDllConfig.outputPath, ...arguments);
}
async readEntryFile() {
return await this.readFile(this.getEntryPath());
}
async readFile(filePath, content) {
await this.ensureFileExists(filePath, content);
return await readFileAsync(filePath, 'utf8');
}
async run(dllEntries) {
const dllConfig = this.dllConfigGenerator(this.rawDllConfig);
await this.upsertEntryFile(dllEntries);
await this.runWebpack(dllConfig());
// Style dll file isn't always created but we are
// expecting it to exist always as we are referencing
// it from the bootstrap template
//
// NOTE: We should review the way we deal with the css extraction
// in ours webpack builds. The industry standard is about to
// only extract css for production but we are extracting it
// in every single compilation.
await this.ensureStyleFileExists();
}
dllConfigGenerator(dllConfig) {
return configModel.bind(this, dllConfig);
}
async runWebpack(config) {
return new Promise((resolve, reject) => {
this.log(['info', 'optimize:dynamic_dll_plugin'], 'Client vendors dll compilation started');
webpack(config, (err, stats) => {
// If a critical error occurs or we have
// errors in the stats compilation,
// reject the promise and logs the errors
const webpackErrors = err || (stats.hasErrors() && stats.toString({
all: false,
colors: true,
errors: true,
errorDetails: true,
moduleTrace: true
}));
if (webpackErrors) {
this.log(
['fatal', 'optimize:dynamic_dll_plugin'],
`Client vendors dll compilation failed`
);
return reject(webpackErrors);
}
// Otherwise let it proceed
this.log(
['info', 'optimize:dynamic_dll_plugin'],
`Client vendors dll compilation finished with success`
);
return resolve(stats);
});
});
}
}

242
src/optimize/dynamic_dll_plugin/dll_config_model.js Normal file
View file

@ -0,0 +1,242 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import { fromRoot, IS_KIBANA_DISTRIBUTABLE } from '../../utils';
import webpack from 'webpack';
import webpackMerge from 'webpack-merge';
import MiniCssExtractPlugin from 'mini-css-extract-plugin';
import UglifyJsPlugin from 'uglifyjs-webpack-plugin';
function generateDLL(config) {
const {
dllAlias,
dllNoParseRules,
dllContext,
dllEntry,
dllOutputPath,
dllPublicPath,
dllBundleName,
dllBundleFilename,
dllStyleFilename,
dllManifestPath
} = config;
const BABEL_PRESET_PATH = require.resolve('@kbn/babel-preset/webpack_preset');
const BABEL_EXCLUDE_RE = [
/[\/\\](webpackShims|node_modules|bower_components)[\/\\]/,
];
return {
entry: dllEntry,
context: dllContext,
output: {
filename: dllBundleFilename,
path: dllOutputPath,
publicPath: dllPublicPath,
library: dllBundleName
},
node: { fs: 'empty', child_process: 'empty', dns: 'empty', net: 'empty', tls: 'empty' },
resolve: {
extensions: ['.js', '.json'],
mainFields: ['browser', 'browserify', 'main'],
alias: dllAlias,
modules: [
'webpackShims',
fromRoot('webpackShims'),
'node_modules',
fromRoot('node_modules'),
],
},
module: {
rules: [
{
resource: [
{
test: /\.js$/,
exclude: BABEL_EXCLUDE_RE.concat(dllNoParseRules),
},
{
test: /\.js$/,
include: /[\/\\]node_modules[\/\\]x-pack[\/\\]/,
exclude: /[\/\\]node_modules[\/\\]x-pack[\/\\](.+?[\/\\])*node_modules[\/\\]/,
}
],
use: {
loader: 'babel-loader',
options: {
babelrc: false,
presets: [
BABEL_PRESET_PATH,
],
},
}
},
{
test: /\.(html|tmpl)$/,
loader: 'raw-loader'
},
{
test: /\.css$/,
use: [
MiniCssExtractPlugin.loader,
'css-loader'
],
},
{
test: /\.png$/,
loader: 'url-loader'
},
{
test: /\.(woff|woff2|ttf|eot|svg|ico)(\?|$)/,
loader: 'file-loader'
},
],
noParse: dllNoParseRules,
},
plugins: [
new webpack.DllPlugin({
context: dllContext,
name: dllBundleName,
path: dllManifestPath
}),
new MiniCssExtractPlugin({
filename: dllStyleFilename
}),
],
performance: {
// NOTE: we are disabling this as those hints
// are more tailored for the final bundles result
// and not for the webpack compilations performance itself
hints: false
}
};
}
function extendRawConfig(rawConfig) {
// Build all extended configs from raw config
const dllAlias = rawConfig.alias;
const dllNoParseRules = rawConfig.noParseRules;
const dllContext = rawConfig.context;
const dllEntry = {};
const dllEntryName = rawConfig.entryName;
const dllBundleName = rawConfig.dllName;
const dllManifestName = rawConfig.dllName;
const dllStyleName = rawConfig.styleName;
const dllEntryExt = rawConfig.entryExt;
const dllBundleExt = rawConfig.dllExt;
const dllManifestExt = rawConfig.manifestExt;
const dllStyleExt = rawConfig.styleExt;
const dllOutputPath = rawConfig.outputPath;
const dllPublicPath = rawConfig.publicPath;
const dllBundleFilename = `${dllBundleName}${dllBundleExt}`;
const dllManifestPath = `${dllOutputPath}/${dllManifestName}${dllManifestExt}`;
const dllStyleFilename = `${dllStyleName}${dllStyleExt}`;
// Create webpack entry object key with the provided dllEntryName
dllEntry[dllEntryName] = [
`${dllOutputPath}/${dllEntryName}${dllEntryExt}`
];
// Export dll config map
return {
dllAlias,
dllNoParseRules,
dllContext,
dllEntry,
dllOutputPath,
dllPublicPath,
dllBundleName,
dllBundleFilename,
dllStyleFilename,
dllManifestPath
};
}
function common(rawConfig) {
return webpackMerge(
generateDLL(extendRawConfig(rawConfig))
);
}
function optimized() {
return webpackMerge(
{
mode: 'production',
optimization: {
minimize: true,
minimizer: [
new UglifyJsPlugin({
parallel: true,
sourceMap: false,
uglifyOptions: {
compress: {
// The following is required for dead-code the removal
// check in React DevTools
//
// default
unused: true,
dead_code: true,
conditionals: true,
evaluate: true,
// changed
keep_fnames: true,
keep_infinity: true,
comparisons: false,
sequences: false,
properties: false,
drop_debugger: false,
booleans: false,
loops: false,
toplevel: false,
top_retain: false,
hoist_funs: false,
if_return: false,
join_vars: false,
collapse_vars: false,
reduce_vars: false,
warnings: false,
negate_iife: false,
side_effects: false
},
mangle: false
}
}),
]
}
}
);
}
function unoptimized() {
return webpackMerge(
{
mode: 'development'
}
);
}
export function configModel(rawConfig = {}) {
if (IS_KIBANA_DISTRIBUTABLE) {
return webpackMerge(common(rawConfig), optimized());
}
return webpackMerge(common(rawConfig), unoptimized());
}

467
src/optimize/dynamic_dll_plugin/dynamic_dll_plugin.js Normal file
View file

@ -0,0 +1,467 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import { DllCompiler } from './dll_compiler';
import { IS_KIBANA_DISTRIBUTABLE } from '../../utils';
import RawModule from 'webpack/lib/RawModule';
import webpack from 'webpack';
import path from 'path';
import normalizePosixPath from 'normalize-path';
import fs from 'fs';
import { promisify } from 'util';
import { parseSingleFileSync, dependenciesVisitorsGenerator } from '@kbn/babel-code-parser';
const realPathAsync = promisify(fs.realpath);
const DLL_ENTRY_STUB_MODULE_TYPE = 'javascript/dll-entry-stub';
function inNodeModulesOrWebpackShims(checkPath) {
return checkPath.includes(`${path.sep}node_modules${path.sep}`)
|| checkPath.includes(`${path.sep}webpackShims${path.sep}`);
}
function inPluginNodeModules(checkPath) {
return checkPath.match(/[\/\\]plugins.*[\/\\]node_modules/);
}
export class DynamicDllPlugin {
constructor({ uiBundles, log, maxCompilations = 1 }) {
this.log = log || (() => null);
this.dllCompiler = new DllCompiler(uiBundles, log);
this.entryPaths = '';
this.afterCompilationEntryPaths = '';
this.maxCompilations = maxCompilations;
this.performedCompilations = 0;
this.forceDLLCreationFlag = !!(process && process.env && process.env.FORCE_DLL_CREATION);
}
async init() {
await this.dllCompiler.init();
this.entryPaths = await this.dllCompiler.readEntryFile();
}
apply(compiler) {
// Just register the init basic hooks
// in order to run the init function
this.registerInitBasicHooks(compiler);
// The dll reference should always be bind to the
// main webpack config.
this.bindDllReferencePlugin(compiler);
// Verify if we must init and run the dynamic dll plugin tasks.
// We must run it every time we are not under a distributable env
if (!this.mustRunDynamicDllPluginTasks()) {
return;
}
// This call init all the DynamicDllPlugin tasks
// as it attaches the plugin to the main webpack
// lifecycle hooks needed to perform the logic
this.registerTasksHooks(compiler);
}
bindDllReferencePlugin(compiler) {
const rawDllConfig = this.dllCompiler.rawDllConfig;
const dllContext = rawDllConfig.context;
const dllManifestPath = this.dllCompiler.getManifestPath();
new webpack.DllReferencePlugin({
context: dllContext,
manifest: dllManifestPath,
}).apply(compiler);
}
registerInitBasicHooks(compiler) {
this.registerRunHook(compiler);
this.registerWatchRunHook(compiler);
}
registerTasksHooks(compiler) {
this.log(['info', 'optimize:dynamic_dll_plugin'], 'Started dynamic dll plugin tasks');
this.registerBeforeCompileHook(compiler);
this.registerCompilationHook(compiler);
this.registerDoneHook(compiler);
}
registerRunHook(compiler) {
compiler.hooks.run.tapPromise('DynamicDllPlugin', async () => {
await this.init();
});
}
registerWatchRunHook(compiler) {
compiler.hooks.watchRun.tapPromise('DynamicDllPlugin', async () => {
await this.init();
});
}
registerBeforeCompileHook(compiler) {
compiler.hooks.beforeCompile.tapPromise('DynamicDllPlugin', async ({ normalModuleFactory }) => {
normalModuleFactory.hooks.factory.tap(
'DynamicDllPlugin',
(actualFactory) => (params, cb) => {
// This is used in order to avoid the cache for DLL modules
// resolved from other dependencies
normalModuleFactory.cachePredicate = (module) => !(module.stubType === DLL_ENTRY_STUB_MODULE_TYPE);
// Overrides the normalModuleFactory module creation behaviour
// in order to understand the modules we need to add to the DLL
actualFactory(params, (error, module) => {
if (error || !module) {
cb(error, module);
} else {
this.mapNormalModule(module).then(
(m = module) => cb(undefined, m),
error => cb(error)
);
}
});
}
);
});
}
registerCompilationHook(compiler) {
compiler.hooks.compilation.tap('DynamicDllPlugin', compilation => {
compilation.hooks.needAdditionalPass.tap('DynamicDllPlugin', () => {
// Run the procedures in order to execute our dll compilation
// The process is very straightforward in it's conception:
//
// * 1 - loop through every compilation module in order to start building
// the dll entry paths arrays and assume it is the new entry paths
// * 1.1 - start from adding the modules already included into the dll, if any.
// * 1.2 - adding the new discovered stub modules
// * 1.3 - check if the module added to the entry path is from node_modules or
// webpackShims, otherwise throw an error.
// * 1.3.1 - for the entry path modules coming from webpackShims search for every
// require statements inside of them
// * 1.3.2 - discard the ones that are not js dependencies
// * 1.3.3 - add those new discovered dependencies inside the webpackShims to the
// entry paths array
// * 2 - compare the built entry paths and compares it to the old one (if any)
// * 3 - runs a new dll compilation in case there is none old entry paths or if the
// new built one differs from the old one.
//
const rawDllConfig = this.dllCompiler.rawDllConfig;
const dllContext = rawDllConfig.context;
const dllOutputPath = rawDllConfig.outputPath;
const requiresMap = {};
const resolvedShimsDependenciesMap = {};
for (const module of compilation.modules) {
let requiredModulePath = null;
// re-include requires for modules already handled by the dll
if (module.delegateData) {
const absoluteResource = path.resolve(dllContext, module.userRequest);
if (absoluteResource.includes('node_modules') || absoluteResource.includes('webpackShims')) {
// NOTE: normalizePosixPath is been used as we only want to have posix
// paths inside our final dll entry file
requiresMap[`require('${normalizePosixPath(path.relative(dllOutputPath, absoluteResource))}');`] = true;
requiredModulePath = absoluteResource;
}
}
// include requires for modules that need to be added to the dll
if (module.stubType === DLL_ENTRY_STUB_MODULE_TYPE) {
requiresMap[`require('${normalizePosixPath(path.relative(dllOutputPath, module.stubResource))}');`] = true;
requiredModulePath = module.stubResource;
}
// read new requires for modules that reaches the compilation,
// aren't already being handled by dll and were not also
// in the entry paths before. The majority should come
// from webpackShims, otherwise we should throw
if (requiredModulePath && !requiredModulePath.includes('node_modules')) {
if (!requiredModulePath.includes('webpackShims')) {
throw new Error(
`The following module is reaching the compilation and ins\'t either a node_module or webpackShim:
${requiredModulePath}
`
);
}
// Get dependencies found in each webpack shim entry and just
// adds them to the global map for the resolvedShimsDependencies
Object.assign(
resolvedShimsDependenciesMap,
this.getDependenciesFromShim(requiredModulePath, compilation)
);
}
}
// Adds the discovered dep modules in webpackShims
// to the final require results
Object.assign(
requiresMap,
this.getRequireEntriesFromShimsDependencies(resolvedShimsDependenciesMap, dllOutputPath)
);
// Sort and join all the discovered require deps
// in order to create a consistent entry file
this.afterCompilationEntryPaths = Object.keys(requiresMap).sort().join('\n');
// The dll compilation will run if on of the following conditions return true:
// 1 - the new generated entry paths are different from the
// old ones
// 2 - if no dll bundle is yet created
// 3 - if this.forceDLLCreationFlag were set from the node env var FORCE_DLL_CREATION and
// we are not running over the distributable. If we are running under the watch optimizer,
// this.forceDLLCreationFlag will only be applied in the very first execution,
// then will be set to false
compilation.needsDLLCompilation = (this.afterCompilationEntryPaths !== this.entryPaths)
|| !this.dllCompiler.dllExistsSync()
|| (this.isToForceDLLCreation() && this.performedCompilations === 0);
this.entryPaths = this.afterCompilationEntryPaths;
// Only run this info log in the first performed dll compilation
// per each execution run
if (this.performedCompilations === 0) {
this.log(
['info', 'optimize:dynamic_dll_plugin'],
compilation.needsDLLCompilation
? 'Need to compile the client vendors dll'
: 'No need to compile client vendors dll'
);
}
return compilation.needsDLLCompilation;
});
});
}
registerDoneHook(compiler) {
compiler.hooks.done.tapPromise('DynamicDllPlugin', async stats => {
if (stats.compilation.needsDLLCompilation) {
// Logic to run the max compilation requirements.
// Only enable this for CI builds in order to ensure
// we have an healthy dll ecosystem.
if (IS_KIBANA_DISTRIBUTABLE && (this.performedCompilations === this.maxCompilations)) {
throw new Error(
'All the allowed dll compilations were already performed and one more is needed which is not possible'
);
}
// Run the dlls compiler and increment
// the performed compilations
await this.runDLLCompiler(compiler);
this.performedCompilations++;
return;
}
this.performedCompilations = 0;
// reset this flag var set from the node env FORCE_DLL_CREATION on init,
// has the force_dll_creation is only valid for the very first run
if (this.forceDLLCreationFlag) {
this.forceDLLCreationFlag = false;
}
this.log(['info', 'optimize:dynamic_dll_plugin'], 'Finished all dynamic dll plugin tasks');
});
}
getDependenciesFromShim(requiredModulePath, compilation) {
// NOTE: is possible we are able to do this reading and searching
// through the compilation's webpack modules, however
// for a sake of simplicity, and as the webpackShims
// should be really small files, we are parsing them
// manually and getting the requires
// Internal map to keep track of the dependencies found for the
// current webpackShim file
const resolvedShimDependencies = {};
// Discover the requires inside the webpackShims
const shimsDependencies = parseSingleFileSync(requiredModulePath, dependenciesVisitorsGenerator);
// Resolve webpackShims dependencies with alias
shimsDependencies.forEach((dep) => {
const isRelative = dep && dep.charAt(0) === '.';
let absoluteResource = null;
// check if the dependency value is relative
if (isRelative) {
absoluteResource = path.resolve(path.dirname(requiredModulePath), dep);
} else {
// get the imports and search for alias in the dependency
const alias = compilation.compiler.options.resolve.alias;
const aliasFound = Object.keys(alias).find((aliasKey) => {
return dep.search(`${aliasKey}/`) !== -1;
});
// search for imports with webpack-loaders
const webpackLoaderFoundIdx = dep.search('!');
if (webpackLoaderFoundIdx !== -1) {
// get the loader
const loader = dep.substring(0, webpackLoaderFoundIdx);
// get the rest of the dependency require value
// after the webpack loader char (!)
const restImport = dep.substring(webpackLoaderFoundIdx + 1);
// build the first part with the loader resolved
const absoluteResourceFirstPart = require.resolve(loader);
// check if we have a relative path in the script require
// path being passed to the loader
const isRestImportRelative = restImport && restImport.charAt(0) === '.';
// resolve the relative script dependency path
// in case we have one
const sanitizedRestImport = isRestImportRelative
? path.resolve(path.dirname(requiredModulePath), restImport)
: restImport;
// replace the alias in the script dependency require path
// in case we have found the alias
const absoluteResourceSecondPart = aliasFound
? require.resolve(`${alias[aliasFound]}${sanitizedRestImport.substring(aliasFound.length)}`)
: require.resolve(sanitizedRestImport);
// finally build our absolute entry path again in the
// original loader format `webpack-loader!script_path`
absoluteResource = `${absoluteResourceFirstPart}!${absoluteResourceSecondPart}`;
} else {
// in case we don't have any webpack loader in the
// dependency require value, just replace the alias
// if we have one and then resolve the result,
// or just resolve the dependency path if we don't
// have any alias
absoluteResource = aliasFound
? require.resolve(`${alias[aliasFound]}${dep.substring(aliasFound.length)}`)
: require.resolve(dep);
}
}
// Only consider found js entries
if (!absoluteResource.includes('.js') || absoluteResource.includes('json')) {
return;
}
// add the absolute built resource to the list of
// entry paths found inside the webpackShims
// to be merged with the general requiresMap
// in the end
resolvedShimDependencies[absoluteResource] = true;
});
return resolvedShimDependencies;
}
getRequireEntriesFromShimsDependencies(resolvedShimsDependenciesMap, dllOutputPath) {
const internalRequiresMap = {};
const resolvedShimsDependencies = Object.keys(resolvedShimsDependenciesMap);
resolvedShimsDependencies.forEach((resolvedDep) => {
if (resolvedDep) {
// check if this is a require shim dependency with
// an webpack-loader
const webpackLoaderFoundIdx = resolvedDep.search('!');
if (webpackLoaderFoundIdx !== -1) {
// get the webpack-loader
const loader = resolvedDep.substring(0, webpackLoaderFoundIdx);
// get the rest of the dependency require value
// after the webpack-loader char (!)
const restImport = resolvedDep.substring(webpackLoaderFoundIdx + 1);
// resolve the loader and the restImport parts separately
const resolvedDepToRequireFirstPart = normalizePosixPath(path.relative(dllOutputPath, loader));
const resolvedDepToRequireSecondPart = normalizePosixPath(path.relative(dllOutputPath, restImport));
// rebuild our final webpackShim entry path in the original
// webpack loader format `webpack-loader!script_path`
// but right now resolved relatively to the dll output path
internalRequiresMap[`require('${resolvedDepToRequireFirstPart}!${resolvedDepToRequireSecondPart}');`] = true;
} else {
// in case we didn't have any webpack-loader in the require path
// resolve the dependency path relative to the dllOutput path
// to get our final entry path
internalRequiresMap[`require('${normalizePosixPath(path.relative(dllOutputPath, resolvedDep))}');`] = true;
}
}
});
return internalRequiresMap;
}
isToForceDLLCreation() {
return this.forceDLLCreationFlag;
}
mustRunDynamicDllPluginTasks() {
return !IS_KIBANA_DISTRIBUTABLE || this.isToForceDLLCreation();
}
async mapNormalModule(module) {
// ignore anything that doesn't have a resource (ignored) or is already delegating to the DLL
if (!module.resource || module.delegateData) {
return;
}
// ignore anything that needs special loaders or config
if (module.request.includes('!') || module.request.includes('?')) {
return;
}
// ignore files that are not in node_modules
if (!inNodeModulesOrWebpackShims(module.resource)) {
return;
}
// also ignore files that are symlinked into node_modules, but only
// do the `realpath` call after checking the plain resource path
if (!inNodeModulesOrWebpackShims(await realPathAsync(module.resource))) {
return;
}
const dirs = module.resource.split(path.sep);
const nodeModuleName = dirs[dirs.lastIndexOf('node_modules') + 1];
// ignore webpack loader modules
if (nodeModuleName.endsWith('-loader')) {
return;
}
// ignore modules from plugins
if (inPluginNodeModules(module.resource)) {
return;
}
// also ignore files that are symlinked into plugins node_modules, but only
// do the `realpath` call after checking the plain resource path
if (inPluginNodeModules(await realPathAsync(module.resource))) {
return;
}
// This is a StubModule (as a RawModule) in order
// to mimic the missing modules from the DLL and
// also hold useful metadata
const stubModule = new RawModule(
`/* pending dll entry */`,
`dll pending:${module.resource}`,
module.resource
);
stubModule.stubType = DLL_ENTRY_STUB_MODULE_TYPE;
stubModule.stubResource = module.resource;
stubModule.stubOriginalModule = module;
return stubModule;
}
async runDLLCompiler(mainCompiler) {
await this.dllCompiler.run(this.entryPaths);
// We need to purge the cache into the inputFileSystem
// for every single built in previous compilation
// that we rely in next ones.
mainCompiler.inputFileSystem.purge(this.dllCompiler.getManifestPath());
}
}

21
src/optimize/dynamic_dll_plugin/index.js Normal file
View file

@ -0,0 +1,21 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
export { DynamicDllPlugin } from './dynamic_dll_plugin';
export { DllCompiler } from './dll_compiler';

10
src/optimize/fs_optimizer.js
View file

@ -21,18 +21,16 @@ import BaseOptimizer from './base_optimizer';
import { fromNode } from 'bluebird';
export default class FsOptimizer extends BaseOptimizer {
async init() {
await this.initCompiler();
}
async run() {
if (!this.compiler) await this.init();
if (!this.compiler) {
await this.init();
}
await fromNode(cb => {
this.compiler.run((err, stats) => {
if (err || !stats) return cb(err);
if (stats.hasErrors() || stats.hasWarnings()) {
if (this.isFailure(stats)) {
return cb(this.failedStatsToError(stats));
}
else {

12
src/optimize/index.js
View file

@ -19,6 +19,7 @@
import FsOptimizer from './fs_optimizer';
import { createBundlesRoute } from './bundles_route';
import { DllCompiler } from './dynamic_dll_plugin';
export default async (kbnServer, server, config) => {
if (!config.get('optimize.enabled')) return;
@ -28,9 +29,9 @@ export default async (kbnServer, server, config) => {
// bundles in a "middleware" style.
//
// the server listening on 5601 may be restarted a number of times, depending
// on the watch setup managed by the cli. It proxies all bundles/* requests to
// the other server. The server on 5602 is long running, in order to prevent
// complete rebuilds of the optimize content.
// on the watch setup managed by the cli. It proxies all bundles/* and dlls/*
// requests to the other server. The server on 5602 is long running, in order
// to prevent complete rebuilds of the optimize content.
const watch = config.get('optimize.watch');
if (watch) {
return await kbnServer.mixin(require('./watch/watch'));
@ -38,7 +39,8 @@ export default async (kbnServer, server, config) => {
const { uiBundles } = kbnServer;
server.route(createBundlesRoute({
bundlesPath: uiBundles.getWorkingDir(),
regularBundlesPath: uiBundles.getWorkingDir(),
dllBundlesPath: DllCompiler.getRawDllConfig().outputPath,
basePublicPath: config.get('server.basePath')
}));
@ -60,10 +62,10 @@ export default async (kbnServer, server, config) => {
// only require the FsOptimizer when we need to
const optimizer = new FsOptimizer({
log: (tags, data) => server.log(tags, data),
uiBundles,
profile: config.get('optimize.profile'),
sourceMaps: config.get('optimize.sourceMaps'),
unsafeCache: config.get('optimize.unsafeCache'),
});
server.log(

3
src/optimize/watch/optmzr_role.js
View file

@ -21,6 +21,7 @@ import { resolve } from 'path';
import WatchServer from './watch_server';
import WatchOptimizer, { STATUS } from './watch_optimizer';
import { DllCompiler } from '../dynamic_dll_plugin';
import { WatchCache } from './watch_cache';
export default async (kbnServer, kibanaHapiServer, config) => {
@ -32,10 +33,10 @@ export default async (kbnServer, kibanaHapiServer, config) => {
profile: config.get('optimize.profile'),
sourceMaps: config.get('optimize.sourceMaps'),
prebuild: config.get('optimize.watchPrebuild'),
unsafeCache: config.get('optimize.unsafeCache'),
watchCache: new WatchCache({
log,
outputPath: config.get('path.data'),
dllsPath: DllCompiler.getRawDllConfig().outputPath,
cachePath: resolve(kbnServer.uiBundles.getCacheDirectory(), '../'),
})
});

20
src/optimize/watch/proxy_role.js
View file

@ -17,24 +17,18 @@
* under the License.
*/
import { createProxyBundlesRoute } from '../bundles_route';
import { fromNode } from 'bluebird';
import { get, once } from 'lodash';
export default (kbnServer, server, config) => {
server.route({
path: '/bundles/{path*}',
method: 'GET',
handler: {
proxy: {
host: config.get('optimize.watchHost'),
port: config.get('optimize.watchPort'),
passThrough: true,
xforward: true
}
},
config: { auth: false }
});
server.route(
createProxyBundlesRoute({
host: config.get('optimize.watchHost'),
port: config.get('optimize.watchPort')
})
);
return fromNode(cb => {
const timeout = setTimeout(() => {

2
src/optimize/watch/watch.js
View file

@ -33,7 +33,7 @@ export default async kbnServer => {
* while the optimizer is running
*
* server: this process runs the entire kibana server and proxies
* all requests for /bundles/* to the optmzr process
* all requests for /bundles/* or /dlls/* to the optmzr process
*
* @param {string} process.env.kbnWorkerType
*/

16
src/optimize/watch/watch_cache.ts
View file

@ -33,6 +33,7 @@ const writeAsync = promisify(writeFile);
interface Params {
log: (tags: string[], data: string) => void;
outputPath: string;
dllsPath: string;
cachePath: string;
}
@ -44,6 +45,7 @@ interface WatchCacheStateContent {
export class WatchCache {
private readonly log: Params['log'];
private readonly outputPath: Params['outputPath'];
private readonly dllsPath: Params['dllsPath'];
private readonly cachePath: Params['cachePath'];
private readonly cacheState: WatchCacheStateContent;
private statePath: string;
@ -53,6 +55,7 @@ export class WatchCache {
constructor(params: Params) {
this.log = params.log;
this.outputPath = params.outputPath;
this.dllsPath = params.dllsPath;
this.cachePath = params.cachePath;
this.isInitialized = false;
@ -107,6 +110,9 @@ export class WatchCache {
// from the previous cache path reset action
await deleteEmpty(this.cachePath);
// delete dlls
await del(this.dllsPath);
// re-write new cache state file
await this.write();
@ -118,7 +124,7 @@ export class WatchCache {
for (const filePath of filePaths) {
try {
shaHash.update(await readAsync(filePath), 'utf8');
shaHash.update(await readAsync(filePath, 'utf8'), 'utf8');
} catch (e) {
/* no-op */
}
@ -135,8 +141,14 @@ export class WatchCache {
private async buildOptimizerConfigSha() {
const baseOptimizer = resolve(__dirname, '../base_optimizer.js');
const dynamicDllConfigModel = resolve(__dirname, '../dynamic_dll_plugin/dll_config_model.js');
const dynamicDllPlugin = resolve(__dirname, '../dynamic_dll_plugin/dynamic_dll_plugin.js');
return await this.buildShaWithMultipleFiles([baseOptimizer]);
return await this.buildShaWithMultipleFiles([
baseOptimizer,
dynamicDllConfigModel,
dynamicDllPlugin,
]);
}
private isResetNeeded() {

89
src/optimize/watch/watch_optimizer.js
View file

@ -17,13 +17,12 @@
* under the License.
*/
import BaseOptimizer from '../base_optimizer';
import { createBundlesRoute } from '../bundles_route';
import { DllCompiler } from '../dynamic_dll_plugin';
import * as Rx from 'rxjs';
import { mergeMap, take } from 'rxjs/operators';
import BaseOptimizer from '../base_optimizer';
import { createBundlesRoute } from '../bundles_route';
export const STATUS = {
RUNNING: 'optimizer running',
SUCCESS: 'optimizer completed successfully',
@ -34,7 +33,6 @@ export const STATUS = {
export default class WatchOptimizer extends BaseOptimizer {
constructor(opts) {
super(opts);
this.log = opts.log || (() => null);
this.prebuild = opts.prebuild || false;
this.watchCache = opts.watchCache;
this.status$ = new Rx.ReplaySubject(1);
@ -50,10 +48,8 @@ export default class WatchOptimizer extends BaseOptimizer {
// log status changes
this.status$.subscribe(this.onStatusChangeHandler);
await this.uiBundles.resetBundleDir();
await this.initCompiler();
await super.init();
this.compiler.plugin('watch-run', this.compilerRunStartHandler);
this.compiler.plugin('done', this.compilerDoneHandler);
this.compiler.watch({ aggregateTimeout: 200 }, this.compilerWatchErrorHandler);
if (this.prebuild) {
@ -63,6 +59,52 @@ export default class WatchOptimizer extends BaseOptimizer {
this.initializing = false;
}
/**
*
* Extends the base_optimizer registerCompilerHooks function
* calling extended function also adding a new register function
*
* It gets called by super.init()
*/
registerCompilerHooks() {
super.registerCompilerHooks();
this.registerCompilerWatchRunHook();
}
registerCompilerWatchRunHook() {
this.compiler.hooks.watchRun.tap('watch_optimizer-watchRun', () => {
this.status$.next({
type: STATUS.RUNNING
});
});
}
registerCompilerDoneHook() {
super.registerCompilerDoneHook();
this.compiler.hooks.done.tap('watch_optimizer-done', stats => {
if (stats.compilation.needAdditionalPass) {
return;
}
this.initialBuildComplete = true;
const seconds = parseFloat((stats.endTime - stats.startTime) / 1000).toFixed(2);
if (this.isFailure(stats)) {
this.status$.next({
type: STATUS.FAILURE,
seconds,
error: this.failedStatsToError(stats)
});
} else {
this.status$.next({
type: STATUS.SUCCESS,
seconds,
});
}
});
}
bindToServer(server, basePath) {
// pause all requests received while the compiler is running
// and continue once an outcome is reached (aborting the request
@ -73,7 +115,8 @@ export default class WatchOptimizer extends BaseOptimizer {
});
server.route(createBundlesRoute({
bundlesPath: this.compiler.outputPath,
regularBundlesPath: this.compiler.outputPath,
dllBundlesPath: DllCompiler.getRawDllConfig().outputPath,
basePublicPath: basePath
}));
}
@ -100,14 +143,6 @@ export default class WatchOptimizer extends BaseOptimizer {
}
}
compilerRunStartHandler = (watchingCompiler, cb) => {
this.status$.next({
type: STATUS.RUNNING
});
cb();
}
compilerWatchErrorHandler = (error) => {
if (error) {
this.status$.next({
@ -115,25 +150,7 @@ export default class WatchOptimizer extends BaseOptimizer {
error
});
}
}
compilerDoneHandler = (stats) => {
this.initialBuildComplete = true;
const seconds = parseFloat((stats.endTime - stats.startTime) / 1000).toFixed(2);
if (stats.hasErrors() || stats.hasWarnings()) {
this.status$.next({
type: STATUS.FAILURE,
seconds,
error: this.failedStatsToError(stats)
});
} else {
this.status$.next({
type: STATUS.SUCCESS,
seconds,
});
}
}
};
onStatusChangeHandler = ({ type, seconds, error }) => {
switch (type) {

11
src/server/config/schema.js
View file

@ -220,17 +220,6 @@ export default () => Joi.object({
watchPrebuild: Joi.boolean().default(false),
watchProxyTimeout: Joi.number().default(5 * 60000),
useBundleCache: Joi.boolean().default(Joi.ref('$prod')),
unsafeCache: Joi.when('$prod', {
is: true,
then: Joi.boolean().valid(false),
otherwise: Joi
.alternatives()
.try(
Joi.boolean(),
Joi.string().regex(/^\/.+\/$/)
)
.default(true),
}),
sourceMaps: Joi.when('$prod', {
is: true,
then: Joi.boolean().valid(false),

1
src/server/logging/log_format_string.js
View file

@ -49,6 +49,7 @@ const typeColors = {
optmzr: 'white',
manager: 'green',
optimize: 'magentaBright',
'optimize:dynamic_dll_plugin': 'magentaBright',
'optimize:watch_cache': 'magentaBright',
listening: 'magentaBright',
scss: 'magentaBright',

1
src/ui/public/notify/index.d.ts vendored
View file

@ -18,3 +18,4 @@
*/
export { toastNotifications, Toast, ToastInput } from './toasts';
export { fatalError } from './fatal_error';

14
src/ui/public/state_management/__tests__/state.js
View file

@ -293,10 +293,18 @@ describe('State Management', () => {
expect(toastNotifications.list[0].title).to.match(/use the share functionality/i);
});
it('triggers fatal error linking to github when setting item fails', () => {
it.skip('triggers fatal error linking to github when setting item fails', () => {
// NOTE: this test needs to be written in jest and removed from the browser ones
// More info could be read in the opened issue:
// https://github.com/elastic/kibana/issues/22751
const { state, hashedItemStore } = setup({ storeInHash: true });
const fatalErrorStub = sandbox.stub(FatalErrorNS, 'fatalError');
sinon.stub(hashedItemStore, 'setItem').returns(false);
const fatalErrorStub = sandbox.stub();
Object.defineProperty(FatalErrorNS, 'fatalError', {
writable: true,
value: fatalErrorStub
});
sandbox.stub(hashedItemStore, 'setItem').returns(false);
state.toQueryParam();
sinon.assert.calledOnce(fatalErrorStub);
sinon.assert.calledWith(fatalErrorStub, sinon.match(error => (

6
src/ui/ui_render/bootstrap/app_bootstrap.test.js
View file

@ -22,7 +22,7 @@ import { resolve } from 'path';
const mockTemplate = `
{{appId}}
{{bundlePath}}
{{regularBundlePath}}
{{i18n 'foo' '{"defaultMessage": "bar"}'}}
`;
@ -102,7 +102,7 @@ describe('ui_render/AppBootstrap', () => {
...mockConfig(),
templateData: {
appId: 'not123',
bundlePath: 'not/foo/bar'
regularBundlePath: 'not/foo/bar'
}
};
const bootstrap2 = new AppBootstrap(config2);
@ -119,7 +119,7 @@ function mockConfig() {
return {
templateData: {
appId: 123,
bundlePath: '/foo/bar'
regularBundlePath: '/foo/bar'
}
};
}

6
src/ui/ui_render/bootstrap/template.js.hbs
View file

@ -1,8 +1,8 @@
window.onload = function () {
var files = [
'{{bundlePath}}/vendors.bundle.js',
'{{bundlePath}}/commons.bundle.js',
'{{bundlePath}}/{{appId}}.bundle.js'
'{{dllBundlePath}}/vendors.bundle.dll.js',
'{{regularBundlePath}}/commons.bundle.js',
'{{regularBundlePath}}/{{appId}}.bundle.js'
];
var failure = function () {

12
src/ui/ui_render/ui_render_mixin.js
View file

@ -62,17 +62,19 @@ export function uiRenderMixin(kbnServer, server, config) {
}
const basePath = config.get('server.basePath');
const bundlePath = `${basePath}/bundles`;
const regularBundlePath = `${basePath}/bundles`;
const dllBundlePath = `${basePath}/dlls`;
const styleSheetPaths = [
`${bundlePath}/vendors.style.css`,
`${bundlePath}/commons.style.css`,
`${bundlePath}/${app.getId()}.style.css`,
`${dllBundlePath}/vendors.style.dll.css`,
`${regularBundlePath}/commons.style.css`,
`${regularBundlePath}/${app.getId()}.style.css`,
].concat(kbnServer.uiExports.styleSheetPaths.map(path => `${basePath}/${path.publicPath}`).reverse());
const bootstrap = new AppBootstrap({
templateData: {
appId: app.getId(),
bundlePath,
regularBundlePath,
dllBundlePath,
styleSheetPaths,
}
});

13
tasks/config/karma.js
View file

@ -85,16 +85,17 @@ module.exports = function (grunt) {
// list of files / patterns to load in the browser
files: [
'http://localhost:5610/bundles/vendors.bundle.js',
'http://localhost:5610/dlls/vendors.bundle.dll.js',
'http://localhost:5610/bundles/tests.bundle.js',
'http://localhost:5610/bundles/vendors.style.css',
'http://localhost:5610/dlls/vendors.style.dll.css',
'http://localhost:5610/bundles/tests.style.css'
],
proxies: {
'/tests/': 'http://localhost:5610/tests/',
'/bundles/': 'http://localhost:5610/bundles/'
'/bundles/': 'http://localhost:5610/bundles/',
'/dlls/': 'http://localhost:5610/dlls/'
},
client: {
@ -175,12 +176,10 @@ module.exports = function (grunt) {
singleRun: true,
options: {
files: [
'http://localhost:5610/bundles/vendors.bundle.js',
'http://localhost:5610/bundles/commons.bundle.js',
'http://localhost:5610/dlls/vendors.bundle.dll.js',
`http://localhost:5610/bundles/tests.bundle.js?shards=${TOTAL_CI_SHARDS}&shard_num=${n}`,
'http://localhost:5610/bundles/vendors.style.css',
'http://localhost:5610/bundles/commons.style.css',
'http://localhost:5610/dlls/vendors.style.dll.css',
'http://localhost:5610/bundles/tests.style.css'
]
}

4
test/plugin_functional/plugins/custom_visualizations/package.json
View file

@ -1,4 +0,0 @@
{
"name": "custom_visualziations",
"version": "kibana"
}

0
test/plugin_functional/plugins/custom_visualizations/README.md → test/plugin_functional/plugins/kbn_tp_custom_visualizations/README.md
View file

2
test/plugin_functional/plugins/custom_visualizations/index.js → test/plugin_functional/plugins/kbn_tp_custom_visualizations/index.js
View file

@ -21,7 +21,7 @@ export default function (kibana) {
return new kibana.Plugin({
uiExports: {
visTypes: [
'plugins/custom_visualziations/self_changing_vis/self_changing_vis',
'plugins/kbn_tp_custom_visualizations/self_changing_vis/self_changing_vis',
],
},
});

13
test/plugin_functional/plugins/kbn_tp_custom_visualizations/package.json Normal file
View file

@ -0,0 +1,13 @@
{
"name": "kbn_tp_custom_visualizations",
"version": "1.0.0",
"kibana": {
"version": "kibana",
"templateVersion": "1.0.0"
},
"license": "Apache-2.0",
"dependencies": {
"@elastic/eui": "5.0.0",
"react": "^16.3.0"
}
}

0
test/plugin_functional/plugins/custom_visualizations/public/self_changing_vis/self_changing_components.js → test/plugin_functional/plugins/kbn_tp_custom_visualizations/public/self_changing_vis/self_changing_components.js
View file

0
test/plugin_functional/plugins/custom_visualizations/public/self_changing_vis/self_changing_editor.js → test/plugin_functional/plugins/kbn_tp_custom_visualizations/public/self_changing_vis/self_changing_editor.js
View file

0
test/plugin_functional/plugins/custom_visualizations/public/self_changing_vis/self_changing_vis.js → test/plugin_functional/plugins/kbn_tp_custom_visualizations/public/self_changing_vis/self_changing_vis.js
View file

2
test/plugin_functional/plugins/sample_app_plugin/index.js → test/plugin_functional/plugins/kbn_tp_sample_app_plugin/index.js
View file

@ -23,7 +23,7 @@ export default function (kibana) {
app: {
title: 'Test Plugin App',
description: 'This is a sample plugin for the functional tests.',
main: 'plugins/sample_app_plugin/app',
main: 'plugins/kbn_tp_sample_app_plugin/app',
}
}
});

9
test/plugin_functional/plugins/kbn_tp_sample_app_plugin/package.json Normal file
View file

@ -0,0 +1,9 @@
{
"name": "kbn_tp_sample_app_plugin",
"version": "1.0.0",
"kibana": {
"version": "kibana",
"templateVersion": "1.0.0"
},
"license": "Apache-2.0"
}

0
test/plugin_functional/plugins/sample_app_plugin/public/app.js → test/plugin_functional/plugins/kbn_tp_sample_app_plugin/public/app.js
View file

4
test/plugin_functional/plugins/sample_panel_action/index.js → test/plugin_functional/plugins/kbn_tp_sample_panel_action/index.js
View file

@ -21,8 +21,8 @@ function samplePanelAction(kibana) {
return new kibana.Plugin({
uiExports: {
contextMenuActions: [
'plugins/sample_panel_action/sample_panel_action',
'plugins/sample_panel_action/sample_panel_link',
'plugins/kbn_tp_sample_panel_action/sample_panel_action',
'plugins/kbn_tp_sample_panel_action/sample_panel_link',
],
},
});

13
test/plugin_functional/plugins/kbn_tp_sample_panel_action/package.json Normal file
View file

@ -0,0 +1,13 @@
{
"name": "kbn_tp_sample_panel_action",
"version": "1.0.0",
"kibana": {
"version": "kibana",
"templateVersion": "1.0.0"
},
"license": "Apache-2.0",
"dependencies": {
"@elastic/eui": "5.0.0",
"react": "^16.3.0"
}
}

0
test/plugin_functional/plugins/sample_panel_action/public/sample_panel_action.js → test/plugin_functional/plugins/kbn_tp_sample_panel_action/public/sample_panel_action.js
View file

0
test/plugin_functional/plugins/sample_panel_action/public/sample_panel_link.js → test/plugin_functional/plugins/kbn_tp_sample_panel_action/public/sample_panel_link.js
View file

4
test/plugin_functional/plugins/visualize_embedding/index.js → test/plugin_functional/plugins/kbn_tp_visualize_embedding/index.js
View file

@ -23,7 +23,7 @@ export default function (kibana) {
app: {
title: 'Embedding Vis',
description: 'This is a sample plugin to test embedding of visualizations',
main: 'plugins/visualize_embedding/app',
main: 'plugins/kbn_tp_visualize_embedding/app',
}
},
@ -31,7 +31,7 @@ export default function (kibana) {
// The following lines copy over some configuration variables from Kibana
// to this plugin. This will be needed when embedding visualizations, so that e.g.
// region map is able to get its configuration.
server.injectUiAppVars('visualize_embedding', async () => {
server.injectUiAppVars('kbn_tp_visualize_embedding', async () => {
return await server.getInjectedUiAppVars('kibana');
});
}

14
test/plugin_functional/plugins/kbn_tp_visualize_embedding/package.json Normal file
View file

@ -0,0 +1,14 @@
{
"name": "kbn_tp_visualize_embedding",
"version": "1.0.0",
"kibana": {
"version": "kibana",
"templateVersion": "1.0.0"
},
"license": "Apache-2.0",
"dependencies": {
"@elastic/eui": "5.0.0",
"react": "^16.3.0",
"react-dom": "^16.3.0"
}
}

0
test/plugin_functional/plugins/visualize_embedding/public/app.js → test/plugin_functional/plugins/kbn_tp_visualize_embedding/public/app.js
View file

0
test/plugin_functional/plugins/visualize_embedding/public/components/main.js → test/plugin_functional/plugins/kbn_tp_visualize_embedding/public/components/main.js
View file

0
test/plugin_functional/plugins/visualize_embedding/public/embedding.js → test/plugin_functional/plugins/kbn_tp_visualize_embedding/public/embedding.js
View file

4
test/plugin_functional/plugins/sample_app_plugin/package.json
View file

@ -1,4 +0,0 @@
{
"name": "sample_app_plugin",
"version": "kibana"
}

8
test/plugin_functional/plugins/sample_panel_action/package.json
View file

@ -1,8 +0,0 @@
{
"name": "sample_panel_action",
"version": "kibana",
"dependencies": {
"@elastic/eui": "0.0.55",
"react": "^16.4.1"
}
}

4
test/plugin_functional/plugins/visualize_embedding/package.json
View file

@ -1,4 +0,0 @@
{
"name": "visualize_embedding",
"version": "kibana"
}

8
x-pack/package.json
View file

@ -21,7 +21,7 @@
}
},
"resolutions": {
"**/@types/node": "8.10.21",
"**/@types/node": "8.10.38",
"@types/react": "16.3.14"
},
"devDependencies": {
@ -29,7 +29,7 @@
"@kbn/es": "1.0.0",
"@kbn/plugin-helpers": "9.0.2",
"@kbn/test": "1.0.0",
"@types/angular": "^1.6.50",
"@types/angular": "1.6.50",
"@types/d3-array": "^1.2.1",
"@types/d3-scale": "^2.0.0",
"@types/d3-shape": "^1.2.2",
@ -108,7 +108,7 @@
"supertest-as-promised": "^4.0.2",
"tmp": "0.0.31",
"tree-kill": "^1.1.0",
"ts-loader": "^3.5.0",
"ts-loader": "^5.2.2",
"typescript": "^3.0.3",
"vinyl-fs": "^3.0.2",
"xml-crypto": "^0.10.1",
@ -141,7 +141,7 @@
"apollo-server-errors": "^2.0.2",
"apollo-server-hapi": "^1.3.6",
"axios": "^0.18.0",
"babel-core": "^6.26.0",
"babel-core": "^6.26.3",
"babel-polyfill": "6.20.0",
"babel-preset-es2015": "^6.24.1",
"babel-register": "^6.26.0",

15
x-pack/plugins/canvas/tasks/helpers/webpack.plugins.js
View file

@ -15,6 +15,7 @@ export function getWebpackConfig({ devtool, watch } = {}) {
watch,
devtool,
mode: 'none',
entry: {
'elements/all': path.join(sourceDir, 'elements/register.js'),
'renderers/all': path.join(sourceDir, 'renderers/register.js'),
@ -36,6 +37,16 @@ export function getWebpackConfig({ devtool, watch } = {}) {
path: buildDir,
filename: '[name].js', // Need long paths here.
libraryTarget: 'umd',
// Note: this is needed due to a not yet resolved bug on
// webpack 4 with umd modules generation.
// For now we have 2 quick workarounds: one is what is implemented
// below another is to change the libraryTarget to commonjs
//
// The issues can be followed on:
// https://github.com/webpack/webpack/issues/6642
// https://github.com/webpack/webpack/issues/6525
// https://github.com/webpack/webpack/issues/6677
globalObject: `(typeof self !== 'undefined' ? self : this)`,
},
resolve: {
@ -44,10 +55,10 @@ export function getWebpackConfig({ devtool, watch } = {}) {
},
plugins: [
function loaderFailHandler() {
function LoaderFailHandlerPlugin() {
// bails on error, including loader errors
// see https://github.com/webpack/webpack/issues/708, which does not fix loader errors
this.plugin('done', function(stats) {
this.hooks.done.tapPromise('LoaderFailHandlerPlugin', async stats => {
if (!stats.hasErrors()) return;
const errorMessage = stats.toString('errors-only');
if (watch) console.error(errorMessage);

2367
yarn.lock
View file

File diff suppressed because it is too large Load diff