github-mirrors/kibana
1
0
Fork 0
mirror of https://github.com/elastic/kibana.git synced 2025-04-24 01:38:56 -04:00
Code Issues Projects Releases Packages Wiki Activity

Kibana build tool (#15055)

Browse source 
* Introduce `kbn`, the Kibana build tool

* yarn kbn

* Make all deps devDeps

* Exclude __fixtures__ folder from Jest to avoid warnings

* Review fixes

* Update readme

* Use 'yarn kbn'

* Consistent rootPath

* Link to kbn tool

* Unsupported URL 'debug help' in contributing guide
This commit is contained in:
Kim Joar Bekkelund 2018-01-24 10:34:52 +01:00 • committed by GitHub
parent fd3fa98017
commit 08e48aa847
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
37 changed files with 38000 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

16
CONTRIBUTING.md
View file

@ -160,12 +160,14 @@ nvm install "$(cat .node-version)"
Install the latest version of [yarn](https://yarnpkg.com).
Install dependencies
Bootstrap Kibana and install all the dependencies
```bash
yarn
yarn kbn bootstrap
```
(You can also run `yarn kbn` to see the other available commands. For more info about this tool, see https://github.com/elastic/kibana/tree/master/packages/kbn-build.)
Start elasticsearch.
```bash
@ -191,6 +193,16 @@ Start the development server.
Now you can point your web browser to https://localhost:5601 and start using Kibana! When running `yarn start`, Kibana will also log that it is listening on port 5603 due to the base path proxy, but you should still access Kibana on port 5601.
#### Unsupported URL Type
If you're installing dependencies and seeing an error that looks something like
```
Unsupported URL Type: link:packages/eslint-config-kibana
```
you're likely running `npm`. To install dependencies in Kibana you need to run `yarn kbn bootstrap`. For more info, see [Setting Up Your Development Environment](#setting-up-your-development-environment) above.
#### Customizing `config/kibana.dev.yml`
The `config/kibana.yml` file stores user configuration directives. Since this file is checked into source control, however, developer preferences can't be saved without the risk of accidentally committing the modified version. To make customizing configuration easier during development, the Kibana CLI will look for a `config/kibana.dev.yml` file if run with the `--dev` flag. This file behaves just like the non-dev version and accepts any of the [standard settings](https://www.elastic.co/guide/en/kibana/current/settings.html).

5
package.json
View file

@ -41,6 +41,7 @@
"Yuri Astrakhan <yuri@elastic.co>"
],
"scripts": {
"kbn": "node scripts/kbn",
"test": "grunt test",
"test:dev": "grunt test:dev",
"test:quick": "grunt test:quick",
@ -220,9 +221,9 @@
"yauzl": "2.7.0"
},
"devDependencies": {
"@elastic/eslint-config-kibana": "0.15.0",
"@elastic/eslint-config-kibana": "link:packages/eslint-config-kibana",
"@elastic/eslint-import-resolver-kibana": "1.0.0",
"@elastic/eslint-plugin-kibana-custom": "1.1.0",
"@elastic/eslint-plugin-kibana-custom": "link:packages/eslint-plugin-kibana-custom",
"angular-mocks": "1.4.7",
"babel-eslint": "8.1.2",
"backport": "2.2.0",

10
packages/kbn-build/.babelrc Normal file
View file

@ -0,0 +1,10 @@
{
"presets": [
"stage-3",
["env", {
"targets": {
"node": "current"
}
}]
]
}

220
packages/kbn-build/README.md Normal file
View file

@ -0,0 +1,220 @@
# `kbn-build` — The Kibana build tool
`kbn-build` is a build/monorepo tool inspired by Lerna, which enables sharing
code between Kibana and Kibana plugins.
To run `kbn-build`, go to Kibana root and run `yarn kbn`.
## Why `kbn-build`?
Long-term we want to get rid of Webpack from production (basically, it's causing
a lot of problems, using a lot of memory and adding a lot of complexity).
Ideally we want each plugin to build its own separate production bundles for
both server and UI. To get there all Kibana plugins (including x-pack) need to
be able to build their production bundles separately from Kibana, which means
they need to be able to depend on code from Kibana without `import`-ing random
files directly from the Kibana source code.
From a plugin perspective there are two different types of Kibana dependencies:
runtime and static dependencies. Runtime dependencies are things that are
instantiated at runtime and that are injected into the plugin, for example
config and elasticsearch clients. Static dependencies are those dependencies
that we want to `import`. `eslint-config-kibana` is one example of this, and
it's actually needed because eslint requires it to be a separate package. But we
also have dependencies like `datemath`, `flot`, `eui` and others that we
control, but where we want to `import` them in plugins instead of injecting them
(because injecting them would be painful to work with). (Btw, these examples
aren't necessarily a part of the Kibana repo today, they are just meant as
examples of code that we might at some point want to include in the repo while
having them be `import`able in Kibana plugins like any other npm package)
Another reason we need static dependencies is that we're starting to introduce
TypeScript into Kibana, and to work nicely with TypeScript across plugins we
need to be able to statically import dependencies. We have for example built an
observable library for Kibana in TypeScript and we need to expose both the
functionality and the TypeScript types to plugins (so other plugins built with
TypeScript can depend on the types for the lib).
However, even though we have multiple packages we don't necessarily want to
`npm publish` them. The ideal solution for us is being able to work on code
locally in the Kibana repo and have a nice workflow that doesn't require
publishing, but where we still get the value of having "packages" that are
available to plugins, without these plugins having to import files directly from
the Kibana folder.
Basically, we just want to be able to share "static code" (aka being able to
`import`) between Kibana and Kibana plugins. To get there we need tooling.
`kbn-build` is a tool that helps us manage these static dependencies, and it
enables us to share these packages between Kibana and Kibana plugins. It also
enables these packages to have their own dependencies and their own build
scripts, while still having a nice developer experience.
## How it works
The approach we went for to handle multiple packages in Kibana is relying on
`link:` style dependencies in Yarn. With `link:` dependencies you specify the
relative location to a package instead of a version when adding it to
`package.json`. For example:
```
"eslint-config-kibana": "link:packages/eslint-config-kibana"
```
Now when you run `yarn` it will set up a symlink to this folder instead of
downloading code from the npm registry. That means you can make changes to
eslint-config-kibana and immediately have them available in Kibana itself. No
`npm publish` needed anymore — Kibana will always rely directly on the code
that's in the local packages. And we can also do the same in x-pack-kibana or
any other Kibana plugin, e.g.
```
"eslint-config-kibana": "link:../../kibana/packages/eslint-config-kibana"
```
This works because we moved to a strict location of Kibana plugins,
`../kibana-extra/{pluginName}` relative to Kibana. This is one of the reasons we
wanted to move towards a setup that looks like this:
```
elastic
├── kibana
└── kibana-extra
├── kibana-canvas
└── x-pack-kibana
```
Relying on `link:` style dependencies means we no longer need to `npm publish`
our Kibana specific packages. It also means that plugin authors no longer need
to worry about the versions of the Kibana packages, as they will always use the
packages from their local Kibana.
## The `kbn` use-cases
### Bootstrapping
Now, instead of installing all the dependencies with just running `yarn` you use
the `kbn-build` tool, which can install dependencies (and set up symlinks) in
all the packages using one command (aka "bootstrap" the setup).
To bootstrap Kibana:
```
yarn kbn bootstrap
```
You can specify additional arguments to `yarn`, e.g.
```
yarn kbn bootstrap -- --frozen-lockfile
```
By default, `kbn-build` will bootstrap all packages within Kibana, plus all
Kibana plugins located in `../kibana-extra`. There are several options for
skipping parts of this, e.g. to skip bootstrapping of Kibana plugins:
```
yarn kbn bootstrap --skip-kibana-extra
```
For more details, run:
```
yarn kbn
```
### Running scripts
Some times you want to run the same script across multiple packages and plugins,
e.g. `build` or `test`. Instead of jumping into each package and running
`yarn build` you can run:
```
yarn kbn run build
```
And if needed, you can skip packages in the same way as for bootstrapping, e.g.
`--skip-kibana` and `--skip-kibana-extra`:
```
yarn kbn run build --skip-kibana
```
## Development
This package is run from Kibana root, using `yarn kbn`. This will run the
"pre-built" (aka built and committed to git) version of this tool, which is
located in the `dist/` folder.
If you need to build a new version of this package, run `yarn build` in this
folder.
Even though this file is generated we commit it to Kibana, because it's used
_before_ dependencies are fetched (as this is the tool actually responsible for
fetching dependencies).
## Technical decisions
### Why our own tool?
While exploring the approach to static dependencies we built PoCs using npm 5
(which symlinks packages using [`file:` dependencies][npm5-file]), [Yarn
workspaces][yarn-workspaces], Yarn (using `link:` dependencies), and
[Lerna][lerna].
In the end we decided to build our own tool, based on Yarn and `link:`
dependencies. This gave us the control we wanted, and it fits nicely into our
context (e.g. where publishing to npm isn't necessarily something we want to
do).
### Some notes from this exploration
#### `file:` dependencies in npm<5 and in yarn
When you add a dependency like `"foo": "file:../../kibana/packages/foo"`, both
npm<5 and yarn copies the files into the `node_modules` folder. This means you
can't easily make changes to the plugin while developing. Therefore this is a
no-go.
#### `file:` dependencies in npm5
In npm5 `file:` dependencies changed to symlink instead of copy the files. This
means you can have a nicer workflow while developing packages locally. However,
we hit several bugs when using this feature, and we often had to re-run
`npm install` in packages. This is likely because we used an early version of
the new `file:` dependencies in npm5.
#### `link:` dependencies in Yarn
This is the same feature as `file:` dependencies in npm5. However, we did not
hit any problems with them during our exploration.
#### Yarn workspaces
Enables specifying multiple "workspaces" (aka packages/projects) in
`package.json`. When running `yarn` from the root, Yarn will install all the
dependencies for these workspaces and hoist the dependencies to the root (to
"deduplicate" packages). However:
> Workspaces must be children of the workspace root in term of folder hierarchy.
> You cannot and must not reference a workspace that is located outside of this
> filesystem hierarchy.
So Yarn workspaces requires a shared root, which (at least currently) doesn't
fit Kibana, and it's therefore a no-go for now.
#### Lerna
Lerna is based on symlinking packages (similarly to the [`link`][npm-link]
feature which exists in both npm and Yarn, but it's not directly using that
feature). It's a tool built specifically for managing JavaScript projects with
multiple packages. However, it's primarily built (i.e. optimized) for monorepo
_libraries_, so it's focused on publishing packages and other use-cases that are
not necessarily optimized for our use-cases. It's also not ideal for the setup
we currently have, with one app that "owns everything" and the rest being
packages for that app.
[npm-link]: https://docs.npmjs.com/cli/link
[npm5-file]: https://github.com/npm/npm/pull/15900
[yarn-workspaces]: https://yarnpkg.com/lang/en/docs/workspaces/
[lerna]: https://github.com/lerna/lerna

1
packages/kbn-build/cli.js Normal file
View file

@ -0,0 +1 @@
require('./dist/cli').run(process.argv.slice(2));

33869
packages/kbn-build/dist/cli.js vendored Normal file
View file

File diff suppressed because one or more lines are too long

33
packages/kbn-build/package.json Normal file
View file

@ -0,0 +1,33 @@
{
"name": "@elastic/kbn-build",
"version": "1.0.0",
"license": "Apache-2.0",
"private": true,
"scripts": {
"build": "webpack",
"prettier": "prettier --single-quote --write './src/**/*.js'"
},
"devDependencies": {
"babel-core": "^6.26.0",
"babel-loader": "^7.1.2",
"babel-preset-env": "^1.6.1",
"babel-preset-stage-3": "^6.24.1",
"chalk": "^2.3.0",
"dedent": "^0.7.0",
"del": "^3.0.0",
"execa": "^0.8.0",
"getopts": "^2.0.0",
"glob": "^7.1.2",
"indent-string": "^3.2.0",
"log-symbols": "^2.1.0",
"ora": "^1.3.0",
"pify": "^3.0.0",
"prettier": "^1.9.1",
"read-pkg": "^3.0.0",
"spawn-sync": "^1.0.15",
"string-replace-loader": "^1.3.0",
"strong-log-transformer": "^1.0.6",
"webpack": "^3.10.0",
"wrap-ansi": "^3.0.1"
}
}

63
packages/kbn-build/src/cli.js Normal file
View file

@ -0,0 +1,63 @@
import getopts from 'getopts';
import dedent from 'dedent';
import chalk from 'chalk';
import { resolve } from 'path';
import * as commands from './commands';
import { runCommand } from './run';
function help() {
const availableCommands = Object.keys(commands)
.map(commandName => commands[commandName])
.map(command => `${command.name} - ${command.description}`);
console.log(dedent`
usage: kbn <command> [<args>]
By default commands are run for Kibana itself, all packages in the 'packages/'
folder and for all plugins in '../kibana-extra'.
Available commands:
${availableCommands.join('\n ')}
Global options:
--skip-kibana Do not include the root Kibana project when running command.
--skip-kibana-extra Filter all plugins in ../kibana-extra when running command.
`);
}
export async function run(argv) {
const options = getopts(argv, {
alias: {
h: 'help'
}
});
const args = options._;
if (options.help || args.length === 0) {
help();
return;
}
// This `rootPath` is relative to `./dist/` as that's the location of the
// built version of this tool.
const rootPath = resolve(__dirname, '../../../');
const commandName = args[0];
const extraArgs = args.slice(1);
const commandOptions = { options, extraArgs, rootPath };
const command = commands[commandName];
if (command === undefined) {
console.log(
chalk.red(`[${commandName}] is not a valid command, see 'kbn --help'`)
);
process.exit(1);
}
await runCommand(command, commandOptions);
}

20
packages/kbn-build/src/commands/bootstrap.js vendored Normal file
View file

@ -0,0 +1,20 @@
import chalk from 'chalk';
import { topologicallyBatchProjects } from '../utils/projects';
export const name = 'bootstrap';
export const description = 'Install dependencies and crosslink projects';
export async function run(projects, projectGraph, { extraArgs }) {
const batchedProjects = topologicallyBatchProjects(projects, projectGraph);
console.log(chalk.bold('\nRunning installs in topological order'));
for (const batch of batchedProjects) {
for (const project of batch) {
if (project.hasDependencies()) {
await project.installDependencies({ extraArgs });
}
}
}
}

35
packages/kbn-build/src/commands/clean.js Normal file
View file

@ -0,0 +1,35 @@
import del from 'del';
import chalk from 'chalk';
import { relative } from 'path';
import ora from 'ora';
import { isDirectory } from '../utils/fs';
export const name = 'clean';
export const description =
'Remove the node_modules and target directories from all projects.';
export async function run(projects, projectGraph, { rootPath }) {
const directoriesToDelete = [];
for (const project of projects.values()) {
if (await isDirectory(project.nodeModulesLocation)) {
directoriesToDelete.push(project.nodeModulesLocation);
}
if (await isDirectory(project.targetLocation)) {
directoriesToDelete.push(project.targetLocation);
}
}
if (directoriesToDelete.length === 0) {
console.log(chalk.bold.green('\n\nNo directories to delete'));
} else {
console.log(chalk.bold.red('\n\nDeleting directories:\n'));
for (const dir of directoriesToDelete) {
const deleting = del(dir, { force: true });
ora.promise(deleting, relative(rootPath, dir));
await deleting;
}
}
}

7
packages/kbn-build/src/commands/index.js Normal file
View file

@ -0,0 +1,7 @@
import * as bootstrap from './bootstrap';
import * as clean from './clean';
import * as run from './run';
export { bootstrap };
export { clean };
export { run };

43
packages/kbn-build/src/commands/run.js Normal file
View file

@ -0,0 +1,43 @@
import chalk from 'chalk';
import { topologicallyBatchProjects } from '../utils/projects';
export const name = 'run';
export const description =
'Run script defined in package.json in each package that contains that script.';
export async function run(projects, projectGraph, { extraArgs }) {
const batchedProjects = topologicallyBatchProjects(projects, projectGraph);
if (extraArgs.length === 0) {
console.log(chalk.red.bold('\nNo script specified'));
process.exit(1);
}
const scriptName = extraArgs[0];
const scriptArgs = extraArgs.slice(1);
console.log(
chalk.bold(
`\nRunning script [${chalk.green(
scriptName
)}] in batched topological order\n`
)
);
await parallelizeBatches(batchedProjects, pkg => {
if (pkg.hasScript(scriptName)) {
return pkg.runScriptStreaming(scriptName, scriptArgs);
}
});
}
async function parallelizeBatches(batchedProjects, fn) {
for (const batch of batchedProjects) {
const running = batch.map(pkg => fn(pkg));
// We need to make sure the entire batch has completed before we can move on
// to the next batch
await Promise.all(running);
}
}

21
packages/kbn-build/src/config.js Normal file
View file

@ -0,0 +1,21 @@
import { resolve } from 'path';
/**
* Returns all the paths where plugins are located
*/
export function getProjectPaths(rootPath, options) {
const skipKibanaExtra = Boolean(options['skip-kibana-extra']);
const skipKibana = Boolean(options['skip-kibana']);
const projectPaths = [resolve(rootPath, 'packages/*')];
if (!skipKibana) {
projectPaths.push(rootPath);
}
if (!skipKibanaExtra) {
projectPaths.push(resolve(rootPath, '../kibana-extra/*'));
}
return projectPaths;
}

55
packages/kbn-build/src/run.js Normal file
View file

@ -0,0 +1,55 @@
import chalk from 'chalk';
import wrapAnsi from 'wrap-ansi';
import indentString from 'indent-string';
import { CliError } from './utils/errors';
import { getProjects, buildProjectGraph } from './utils/projects';
import { getProjectPaths } from './config';
export async function runCommand(command, config) {
try {
console.log(
chalk.bold(
`Running [${chalk.green(command.name)}] command from [${chalk.yellow(
config.rootPath
)}]:\n`
)
);
const projectPaths = getProjectPaths(config.rootPath, config.options);
const projects = await getProjects(config.rootPath, projectPaths);
const projectGraph = buildProjectGraph(projects);
console.log(
chalk.bold(`Found [${chalk.green(projects.size)}] projects:\n`)
);
for (const pkg of projects.values()) {
console.log(`- ${pkg.name} (${pkg.path})`);
}
await command.run(projects, projectGraph, config);
} catch (e) {
console.log(chalk.bold.red(`\n[${command.name}] failed:\n`));
if (e instanceof CliError) {
const msg = chalk.red(`CliError: ${e.message}\n`);
console.log(wrapAnsi(msg, 80));
const keys = Object.keys(e.meta);
if (keys.length > 0) {
const metaOutput = keys.map(key => {
const value = e.meta[key];
return `${key}: ${value}`;
});
console.log('Additional debugging info:\n');
console.log(indentString(metaOutput.join('\n'), 3));
}
} else {
console.log(e.stack);
}
process.exit(1);
}
}

7
packages/kbn-build/src/utils/__fixtures__/kibana/package.json Normal file
View file

@ -0,0 +1,7 @@
{
"name": "kibana",
"version": "1.0.0",
"dependencies": {
"foo": "link:packages/foo"
}
}

7
packages/kbn-build/src/utils/__fixtures__/kibana/packages/bar/package.json Normal file
View file

@ -0,0 +1,7 @@
{
"name": "bar",
"version": "1.0.0",
"dependencies": {
"foo": "link:../foo"
}
}

4
packages/kbn-build/src/utils/__fixtures__/kibana/packages/foo/package.json Normal file
View file

@ -0,0 +1,4 @@
{
"name": "foo",
"version": "1.0.0"
}

4
packages/kbn-build/src/utils/__fixtures__/other-plugins/baz/package.json Normal file
View file

@ -0,0 +1,4 @@
{
"name": "baz",
"version": "1.0.0"
}

4
packages/kbn-build/src/utils/__fixtures__/plugins/baz/package.json Normal file
View file

@ -0,0 +1,4 @@
{
"name": "baz",
"version": "1.0.0"
}

8
packages/kbn-build/src/utils/__fixtures__/plugins/quux/package.json Normal file
View file

@ -0,0 +1,8 @@
{
"name": "quux",
"version": "1.0.0",
"dependencies": {
"bar": "link:../../kibana/packages/bar",
"baz": "link:../baz"
}
}

5
packages/kbn-build/src/utils/__snapshots__/project.test.js.snap Normal file
View file

@ -0,0 +1,5 @@
// Jest Snapshot v1, https://goo.gl/fbAQLP
exports[`#ensureValidProjectDependency using link:, but with wrong path 1`] = `"[kibana] depends on [foo] using 'link:', but the path is wrong. Update its package.json to the expected value below."`;
exports[`#ensureValidProjectDependency using version instead of link: 1`] = `"[kibana] depends on [foo], but it's not using the local package. Update its package.json to the expected value below."`;

34
packages/kbn-build/src/utils/__snapshots__/projects.test.js.snap Normal file
View file

@ -0,0 +1,34 @@
// Jest Snapshot v1, https://goo.gl/fbAQLP
exports[`#buildProjectGraph builds full project graph 1`] = `
Object {
"bar": Array [
"foo",
],
"baz": Array [],
"foo": Array [],
"kibana": Array [
"foo",
],
"quux": Array [
"bar",
"baz",
],
}
`;
exports[`#topologicallyBatchProjects batches projects topologically based on their project dependencies 1`] = `
Array [
Array [
"foo",
"baz",
],
Array [
"kibana",
"bar",
],
Array [
"quux",
],
]
`;

42
packages/kbn-build/src/utils/child_process.js Normal file
View file

@ -0,0 +1,42 @@
import execa from 'execa';
import chalk from 'chalk';
import logTransformer from 'strong-log-transformer';
import logSymbols from 'log-symbols';
function generateColors() {
const colorWheel = ['cyan', 'magenta', 'blue', 'yellow', 'green', 'red'].map(
name => chalk[name]
);
const count = colorWheel.length;
let children = 0;
return () => colorWheel[children++ % count];
}
export function spawn(command, args, opts) {
return execa(command, args, {
...opts,
stdio: 'inherit'
});
}
const nextColor = generateColors();
export function spawnStreaming(command, args, opts, { prefix }) {
const spawned = execa(command, args, {
...opts,
stdio: ['ignore', 'pipe', 'pipe']
});
const color = nextColor();
const prefixedStdout = logTransformer({ tag: `${color.bold(prefix)}:` });
const prefixedStderr = logTransformer({
tag: `${logSymbols.error} ${color.bold(prefix)}:`,
mergeMultiline: true
});
spawned.stdout.pipe(prefixedStdout).pipe(process.stdout);
spawned.stderr.pipe(prefixedStderr).pipe(process.stderr);
return spawned;
}

6
packages/kbn-build/src/utils/errors.js Normal file
View file

@ -0,0 +1,6 @@
export class CliError extends Error {
constructor(message, meta = {}) {
super(message);
this.meta = meta;
}
}

16
packages/kbn-build/src/utils/fs.js Normal file
View file

@ -0,0 +1,16 @@
import fs from 'fs';
import promisify from 'pify';
const stat = promisify(fs.stat);
export async function isDirectory(path) {
try {
const targetFolder = await stat(path);
return targetFolder.isDirectory();
} catch (e) {
if (e.code === 'ENOENT') {
return false;
}
throw e;
}
}

6
packages/kbn-build/src/utils/package_json.js Normal file
View file

@ -0,0 +1,6 @@
import readPkg from 'read-pkg';
import path from 'path';
export function readPackageJson(dir) {
return readPkg(path.join(dir, 'package.json'), { normalize: false });
}

90
packages/kbn-build/src/utils/project.js Normal file
View file

@ -0,0 +1,90 @@
import path from 'path';
import chalk from 'chalk';
import { installInDir, runScriptInPackageStreaming } from './scripts';
import { readPackageJson } from './package_json';
import { CliError } from './errors';
const PREFIX = 'link:';
export class Project {
static async fromPath(path) {
const pkgJson = await readPackageJson(path);
return new Project(pkgJson, path);
}
constructor(packageJson, projectPath) {
this._json = packageJson;
this.path = projectPath;
this.packageJsonLocation = path.resolve(this.path, 'package.json');
this.nodeModulesLocation = path.resolve(this.path, 'node_modules');
this.targetLocation = path.resolve(this.path, 'target');
this.allDependencies = {
...(this._json.devDependencies || {}),
...(this._json.dependencies || {})
};
this.scripts = this._json.scripts || {};
}
get name() {
return this._json.name;
}
ensureValidProjectDependency(project) {
const relativePathToProject = path.relative(this.path, project.path);
const versionInPackageJson = this.allDependencies[project.name];
const expectedVersionInPackageJson = `${PREFIX}${relativePathToProject}`;
if (versionInPackageJson === expectedVersionInPackageJson) {
return;
}
const updateMsg = 'Update its package.json to the expected value below.';
const meta = {
package: `${this.name} (${this.packageJsonLocation})`,
expected: `"${project.name}": "${expectedVersionInPackageJson}"`,
actual: `"${project.name}": "${versionInPackageJson}"`
};
if (versionInPackageJson.startsWith(PREFIX)) {
throw new CliError(
`[${this.name}] depends on [${
project.name
}] using '${PREFIX}', but the path is wrong. ${updateMsg}`,
meta
);
}
throw new CliError(
`[${this.name}] depends on [${
project.name
}], but it's not using the local package. ${updateMsg}`,
meta
);
}
hasScript(name) {
return name in this.scripts;
}
runScriptStreaming(scriptName, args = []) {
return runScriptInPackageStreaming(scriptName, args, this);
}
hasDependencies() {
return Object.keys(this.allDependencies).length > 0;
}
installDependencies({ extraArgs }) {
console.log(
chalk.bold(
`\n\nInstalling dependencies in [${chalk.green(this.name)}]:\n`
)
);
return installInDir(this.path, extraArgs);
}
}

105
packages/kbn-build/src/utils/project.test.js Normal file
View file

@ -0,0 +1,105 @@
import { resolve, join } from 'path';
import { Project } from './project';
const rootPath = resolve(`${__dirname}/__fixtures__/kibana`);
const createProjectWith = (fields, path = '') =>
new Project(
{
name: 'kibana',
version: '1.0.0',
...fields
},
join(rootPath, path)
);
describe('fromPath', () => {
test('reads project.json at path and constructs Project', async () => {
const kibana = await Project.fromPath(rootPath);
expect(kibana.name).toBe('kibana');
});
});
test('fields', async () => {
const kibana = createProjectWith({
scripts: {
test: 'jest'
},
dependencies: {
foo: '1.2.3'
}
});
expect(kibana.name).toBe('kibana');
expect(kibana.hasDependencies()).toBe(true);
expect(kibana.allDependencies).toEqual({ foo: '1.2.3' });
expect(kibana.hasScript('test')).toBe(true);
expect(kibana.hasScript('build')).toBe(false);
});
describe('#ensureValidProjectDependency', () => {
test('valid link: version', async () => {
const root = createProjectWith({
dependencies: {
foo: 'link:packages/foo'
}
});
const foo = createProjectWith(
{
name: 'foo'
},
'packages/foo'
);
expect(() => root.ensureValidProjectDependency(foo)).not.toThrow();
});
test('using link:, but with wrong path', () => {
const root = createProjectWith(
{
dependencies: {
foo: 'link:wrong/path'
}
},
rootPath
);
const foo = createProjectWith(
{
name: 'foo'
},
'packages/foo'
);
expect(() =>
root.ensureValidProjectDependency(foo)
).toThrowErrorMatchingSnapshot();
});
test('using version instead of link:', () => {
const root = createProjectWith(
{
dependencies: {
foo: '1.0.0'
}
},
rootPath
);
const foo = createProjectWith(
{
name: 'foo'
},
'packages/foo'
);
expect(() =>
root.ensureValidProjectDependency(foo)
).toThrowErrorMatchingSnapshot();
});
});

125
packages/kbn-build/src/utils/projects.js Normal file
View file

@ -0,0 +1,125 @@
import _glob from 'glob';
import path from 'path';
import promisify from 'pify';
import { CliError } from './errors';
import { Project } from './project';
const glob = promisify(_glob);
export async function getProjects(rootPath, projectsPaths) {
const globOpts = {
cwd: rootPath,
// Should throw in case of unusual errors when reading the file system
strict: true,
// Always returns absolute paths for matched files
absolute: true,
// Do not match ** against multiple filenames
// (This is only specified because we currently don't have a need for it.)
noglobstar: true
};
const projects = new Map();
for (const globPath of projectsPaths) {
const files = await glob(path.join(globPath, 'package.json'), globOpts);
for (const filePath of files) {
const projectConfigPath = normalize(filePath);
const projectDir = path.dirname(projectConfigPath);
const project = await Project.fromPath(projectDir);
if (projects.has(project.name)) {
throw new CliError(
`There are multiple projects with the same name [${project.name}]`,
{
name: project.name,
paths: [project.path, projects.get(project.name).path]
}
);
}
projects.set(project.name, project);
}
}
return projects;
}
// https://github.com/isaacs/node-glob/blob/master/common.js#L104
// glob always returns "\\" as "/" in windows, so everyone
// gets normalized because we can't have nice things.
function normalize(dir) {
return path.normalize(dir);
}
export function buildProjectGraph(projects) {
const projectGraph = new Map();
for (const project of projects.values()) {
const projectDeps = [];
const dependencies = project.allDependencies;
for (const depName of Object.keys(dependencies)) {
if (projects.has(depName)) {
const dep = projects.get(depName);
project.ensureValidProjectDependency(dep);
projectDeps.push(dep);
}
}
projectGraph.set(project.name, projectDeps);
}
return projectGraph;
}
export function topologicallyBatchProjects(projectsToBatch, projectGraph) {
// We're going to be chopping stuff out of this array, so copy it.
const projects = [...projectsToBatch.values()];
// This maps project names to the number of projects that depend on them.
// As projects are completed their names will be removed from this object.
const refCounts = {};
projects.forEach(pkg =>
projectGraph.get(pkg.name).forEach(dep => {
if (!refCounts[dep.name]) refCounts[dep.name] = 0;
refCounts[dep.name]++;
})
);
const batches = [];
while (projects.length > 0) {
// Get all projects that have no remaining dependencies within the repo
// that haven't yet been picked.
const batch = projects.filter(pkg => {
const projectDeps = projectGraph.get(pkg.name);
return projectDeps.filter(dep => refCounts[dep.name] > 0).length === 0;
});
// If we weren't able to find a project with no remaining dependencies,
// then we've encountered a cycle in the dependency graph.
const hasCycles = projects.length > 0 && batch.length === 0;
if (hasCycles) {
const cycleProjectNames = projects.map(p => p.name);
const message =
'Encountered a cycle in the dependency graph. Projects in cycle are:\n' +
cycleProjectNames.join(', ');
throw new CliError(message);
}
batches.push(batch);
batch.forEach(pkg => {
delete refCounts[pkg.name];
projects.splice(projects.indexOf(pkg), 1);
});
}
return batches;
}

86
packages/kbn-build/src/utils/projects.test.js Normal file
View file

@ -0,0 +1,86 @@
import { resolve } from 'path';
import {
getProjects,
buildProjectGraph,
topologicallyBatchProjects
} from './projects';
const rootPath = resolve(`${__dirname}/__fixtures__/kibana`);
describe('#getProjects', () => {
test('find all packages in the packages directory', async () => {
const projects = await getProjects(rootPath, ['packages/*']);
const expectedProjects = ['bar', 'foo'];
expect(projects.size).toBe(2);
expect([...projects.keys()]).toEqual(
expect.arrayContaining(expectedProjects)
);
});
test('can specify root as a separate project', async () => {
const projects = await getProjects(rootPath, ['.']);
expect(projects.size).toBe(1);
expect([...projects.keys()]).toEqual(['kibana']);
});
test('handles packages outside root', async () => {
const projects = await getProjects(rootPath, ['../plugins/*']);
const expectedProjects = ['baz', 'quux'];
expect(projects.size).toBe(2);
expect([...projects.keys()]).toEqual(
expect.arrayContaining(expectedProjects)
);
});
test('throws if multiple projects has the same name', async () => {
await expect(
getProjects(rootPath, ['../plugins/*', '../other-plugins/*'])
).rejects.toHaveProperty(
'message',
'There are multiple projects with the same name [baz]'
);
});
});
describe('#buildProjectGraph', () => {
test('builds full project graph', async () => {
const projects = await getProjects(rootPath, [
'.',
'packages/*',
'../plugins/*'
]);
const graph = buildProjectGraph(projects);
const expected = {};
for (const [projectName, projects] of graph.entries()) {
expected[projectName] = projects.map(project => project.name);
}
expect(expected).toMatchSnapshot();
});
});
describe('#topologicallyBatchProjects', () => {
test('batches projects topologically based on their project dependencies', async () => {
const projects = await getProjects(rootPath, [
'.',
'packages/*',
'../plugins/*'
]);
const graph = buildProjectGraph(projects);
const batches = topologicallyBatchProjects(projects, graph);
const expectedBatches = batches.map(batch =>
batch.map(project => project.name)
);
expect(expectedBatches).toMatchSnapshot();
});
});

32
packages/kbn-build/src/utils/scripts.js Normal file
View file

@ -0,0 +1,32 @@
import { spawn, spawnStreaming } from './child_process';
/**
* Install all dependencies in the given directory
*/
export function installInDir(directory, extraArgs = []) {
const options = [
'install',
'--non-interactive',
'--mutex file',
...extraArgs
];
// We pass the mutex flag to ensure only one instance of yarn runs at any
// given time (e.g. to avoid conflicts).
return spawn('yarn', options, {
cwd: directory
});
}
/**
* Run script in the given directory
*/
export function runScriptInPackageStreaming(script, args, pkg) {
const execOpts = {
cwd: pkg.path
};
return spawnStreaming('yarn', ['run', script, ...args], execOpts, {
prefix: pkg.name
});
}

48
packages/kbn-build/webpack.config.js Normal file
View file

@ -0,0 +1,48 @@
const path = require('path');
module.exports = {
entry: {
cli: './src/cli.js'
},
target: 'node',
output: {
path: path.resolve(__dirname, 'dist'),
filename: '[name].js',
libraryTarget: 'commonjs2',
},
module: {
rules: [
{
test: /\.js$/,
exclude: /(node_modules)/,
use: {
loader: 'babel-loader'
}
},
// Removing an unnecessary require from
// https://github.com/ForbesLindesay/spawn-sync/blob/8ba6d1bd032917ff5f0cf68508b91bb628d16336/index.js#L3
//
// This require would cause warnings when building with Webpack, and it's
// only required for Node <= 0.12.
{
test: /spawn-sync\/index\.js$/,
use: {
loader: 'string-replace-loader',
options: {
search: ` || require('./lib/spawn-sync')`,
replace: '',
strict: true
}
}
}
]
},
node: {
// Don't replace built-in globals
__filename: false,
__dirname: false
}
};

2964
packages/kbn-build/yarn.lock Normal file
View file

File diff suppressed because it is too large Load diff

2
scripts/kbn.js Normal file
View file

@ -0,0 +1,2 @@
require('../packages/kbn-build/cli');

2
src/dev/ci_setup/setup.sh
View file

@ -84,4 +84,4 @@ hash -r
###
echo " -- installing node.js dependencies"
yarn config set cache-folder "$cacheDir/yarn"
yarn --frozen-lockfile
yarn kbn bootstrap -- --frozen-lockfile

6
src/dev/jest/config.json
View file

@ -3,7 +3,8 @@
"roots": [
"<rootDir>/src/ui/public",
"<rootDir>/src/core_plugins",
"<rootDir>/ui_framework/"
"<rootDir>/ui_framework/",
"<rootDir>/packages"
],
"collectCoverageFrom": [
"ui_framework/src/components/**/*.js",
@ -40,7 +41,8 @@
"testPathIgnorePatterns": [
"<rootDir>/ui_framework/dist/",
"<rootDir>/ui_framework/doc_site/",
"<rootDir>/ui_framework/generator-kui/"
"<rootDir>/ui_framework/generator-kui/",
".*/__fixtures__/.*"
],
"transform": {
"^.+\\.js$": "<rootDir>/src/dev/jest/babel_transform.js"

12
yarn.lock
View file

@ -69,9 +69,9 @@
dependencies:
moment "^2.13.0"
"@elastic/eslint-config-kibana@0.15.0":
version "0.15.0"
resolved "https://registry.yarnpkg.com/@elastic/eslint-config-kibana/-/eslint-config-kibana-0.15.0.tgz#a552793497cdfc1829c2f9b7cd7018eb008f1606"
"@elastic/eslint-config-kibana@link:packages/eslint-config-kibana":
version "0.0.0"
uid ""
"@elastic/eslint-import-resolver-kibana@1.0.0":
version "1.0.0"
@ -83,9 +83,9 @@
glob-all "^3.1.0"
webpack "3.6.0"
"@elastic/eslint-plugin-kibana-custom@1.1.0":
version "1.1.0"
resolved "https://registry.yarnpkg.com/@elastic/eslint-plugin-kibana-custom/-/eslint-plugin-kibana-custom-1.1.0.tgz#f4c5a10c16f7a23c46d32be7165e012fa628b967"
"@elastic/eslint-plugin-kibana-custom@link:packages/eslint-plugin-kibana-custom":
version "0.0.0"
uid ""
"@elastic/eui@0.0.13":
version "0.0.13"