mirror of
https://github.com/elastic/kibana.git
synced 2025-06-27 10:40:07 -04:00
Transpile packages on demand, validate all TS projects (#146212)
## Dearest Reviewers 👋 I've been working on this branch with @mistic and @tylersmalley and we're really confident in these changes. Additionally, this changes code in nearly every package in the repo so we don't plan to wait for reviews to get in before merging this. If you'd like to have a concern addressed, please feel free to leave a review, but assuming that nobody raises a blocker in the next 24 hours we plan to merge this EOD pacific tomorrow, 12/22. We'll be paying close attention to any issues this causes after merging and work on getting those fixed ASAP. 🚀 --- The operations team is not confident that we'll have the time to achieve what we originally set out to accomplish by moving to Bazel with the time and resources we have available. We have also bought ourselves some headroom with improvements to babel-register, optimizer caching, and typescript project structure. In order to make sure we deliver packages as quickly as possible (many teams really want them), with a usable and familiar developer experience, this PR removes Bazel for building packages in favor of using the same JIT transpilation we use for plugins. Additionally, packages now use `kbn_references` (again, just copying the dx from plugins to packages). Because of the complex relationships between packages/plugins and in order to prepare ourselves for automatic dependency detection tools we plan to use in the future, this PR also introduces a "TS Project Linter" which will validate that every tsconfig.json file meets a few requirements: 1. the chain of base config files extended by each config includes `tsconfig.base.json` and not `tsconfig.json` 1. the `include` config is used, and not `files` 2. the `exclude` config includes `target/**/*` 3. the `outDir` compiler option is specified as `target/types` 1. none of these compiler options are specified: `declaration`, `declarationMap`, `emitDeclarationOnly`, `skipLibCheck`, `target`, `paths` 4. all references to other packages/plugins use their pkg id, ie: ```js // valid { "kbn_references": ["@kbn/core"] } // not valid { "kbn_references": [{ "path": "../../../src/core/tsconfig.json" }] } ``` 5. only packages/plugins which are imported somewhere in the ts code are listed in `kbn_references` This linter is not only validating all of the tsconfig.json files, but it also will fix these config files to deal with just about any violation that can be produced. Just run `node scripts/ts_project_linter --fix` locally to apply these fixes, or let CI take care of automatically fixing things and pushing the changes to your PR. > **Example:** [`64e93e5` (#146212)](64e93e5806) When I merged main into my PR it included a change which removed the `@kbn/core-injected-metadata-browser` package. After resolving the conflicts I missed a few tsconfig files which included references to the now removed package. The TS Project Linter identified that these references were removed from the code and pushed a change to the PR to remove them from the tsconfig.json files. ## No bazel? Does that mean no packages?? Nope! We're still doing packages but we're pretty sure now that we won't be using Bazel to accomplish the 'distributed caching' and 'change-based tasks' portions of the packages project. This PR actually makes packages much easier to work with and will be followed up with the bundling benefits described by the original packages RFC. Then we'll work on documentation and advocacy for using packages for any and all new code. We're pretty confident that implementing distributed caching and change-based tasks will be necessary in the future, but because of recent improvements in the repo we think we can live without them for **at least** a year. ## Wait, there are still BUILD.bazel files in the repo Yes, there are still three webpack bundles which are built by Bazel: the `@kbn/ui-shared-deps-npm` DLL, `@kbn/ui-shared-deps-src` externals, and the `@kbn/monaco` workers. These three webpack bundles are still created during bootstrap and remotely cached using bazel. The next phase of this project is to figure out how to get the package bundling features described in the RFC with the current optimizer, and we expect these bundles to go away then. Until then any package that is used in those three bundles still needs to have a BUILD.bazel file so that they can be referenced by the remaining webpack builds. Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This commit is contained in:
Spencer
GitHub
parent
e9c5c16bfd
commit
afb09ccf8a
2421 changed files with 17782 additions and 110627 deletions
|
|
@ -38,7 +38,7 @@ There are cases where `@kbn/pm` relies on code from packages, mostly to prevent
|
|||
|
||||
Option 1 is used in several places, with contingencies in place in case bootstrap failed. Option 2 is used for two pieces of code which are needed in order to run bootstrap:
|
||||
|
||||
1. `@kbn/plugin-discovery` as we need to populate the `@kbn/synthetic-package-map` to run Bazel
|
||||
1. `@kbn/plugin-discovery` as we need to populate the `@kbn/package-map` to run Bazel
|
||||
2. `@kbn/bazel-runner` as we want to have the logic for running bazel in a single location
|
||||
|
||||
Because we load these two packages from source, without being built, before bootstrap is ever run, they can not depend on other packages and must be written in Vanilla JS as well.
|
||||
|
|
@ -20,6 +20,7 @@ import { getHelp } from './lib/help.mjs';
|
|||
import { createFlagError, isCliError } from './lib/cli_error.mjs';
|
||||
import { getCmd } from './commands/index.mjs';
|
||||
import { Log } from './lib/log.mjs';
|
||||
import External from './lib/external_packages.js';
|
||||
|
||||
const start = Date.now();
|
||||
const args = new Args(process.argv.slice(2), process.env.CI ? ['--quiet'] : []);
|
||||
|
|
@ -31,7 +32,7 @@ const cmdName = args.getCommandName();
|
|||
*/
|
||||
async function tryToGetCiStatsReporter(log) {
|
||||
try {
|
||||
const { CiStatsReporter } = await import('@kbn/ci-stats-reporter');
|
||||
const { CiStatsReporter } = External['@kbn/ci-stats-reporter']();
|
||||
return CiStatsReporter.fromEnv(log);
|
||||
} catch {
|
||||
return;
|
||||
|
|
|
|||
|
|
@ -8,13 +8,15 @@
|
|||
|
||||
import { run } from '../../lib/spawn.mjs';
|
||||
import * as Bazel from '../../lib/bazel.mjs';
|
||||
import External from '../../lib/external_packages.js';
|
||||
|
||||
import { haveNodeModulesBeenManuallyDeleted, removeYarnIntegrityFileIfExists } from './yarn.mjs';
|
||||
import { setupRemoteCache } from './setup_remote_cache.mjs';
|
||||
import { regenerateSyntheticPackageMap } from './regenerate_synthetic_package_map.mjs';
|
||||
import { sortPackageJson } from './sort_package_json.mjs';
|
||||
import { REPO_ROOT } from '../../lib/paths.mjs';
|
||||
import { pluginDiscovery } from './plugins.mjs';
|
||||
import { regeneratePackageMap } from './regenerate_package_map.mjs';
|
||||
import { regenerateBaseTsconfig } from './regenerate_base_tsconfig.mjs';
|
||||
import { packageDiscovery, pluginDiscovery } from './discovery.mjs';
|
||||
import { validatePackageJson } from './validate_package_json.mjs';
|
||||
|
||||
/** @type {import('../../lib/command').Command} */
|
||||
export const command = {
|
||||
|
|
@ -83,45 +85,50 @@ export const command = {
|
|||
});
|
||||
}
|
||||
|
||||
const plugins = await time('plugin discovery', async () => {
|
||||
return await pluginDiscovery();
|
||||
// discover the location of packages and plugins
|
||||
const [plugins, packages] = await Promise.all([
|
||||
time('plugin discovery', pluginDiscovery),
|
||||
time('package discovery', packageDiscovery),
|
||||
]);
|
||||
|
||||
// generate the package map which powers the resolver and several other features
|
||||
// needed as an input to the bazel builds
|
||||
await time('regenerate package map', async () => {
|
||||
await regeneratePackageMap(packages, plugins, log);
|
||||
});
|
||||
|
||||
// generate the synthetic package map which powers several other features, needed
|
||||
// as an input to the package build
|
||||
await time('regenerate synthetic package map', async () => {
|
||||
await regenerateSyntheticPackageMap(plugins);
|
||||
await time('pre-build webpack bundles for packages', async () => {
|
||||
await Bazel.buildWebpackBundles(log, { offline, quiet });
|
||||
});
|
||||
|
||||
await time('build packages', async () => {
|
||||
await Bazel.buildPackages(log, { offline, quiet });
|
||||
});
|
||||
await time('sort package json', async () => {
|
||||
await sortPackageJson();
|
||||
});
|
||||
await time('regenerate tsconfig.base.json', async () => {
|
||||
const { discoverBazelPackages } = await import('@kbn/bazel-packages');
|
||||
await regenerateBaseTsconfig(await discoverBazelPackages(REPO_ROOT), plugins);
|
||||
await regenerateBaseTsconfig();
|
||||
});
|
||||
|
||||
if (validate) {
|
||||
// now that packages are built we can import `@kbn/yarn-lock-validator`
|
||||
const { readYarnLock, validateDependencies } = await import('@kbn/yarn-lock-validator');
|
||||
const yarnLock = await time('read yarn.lock', async () => {
|
||||
return await readYarnLock();
|
||||
});
|
||||
await time('validate dependencies', async () => {
|
||||
await validateDependencies(log, yarnLock);
|
||||
});
|
||||
}
|
||||
await Promise.all([
|
||||
time('sort package json', async () => {
|
||||
await sortPackageJson();
|
||||
}),
|
||||
time('validate package json', async () => {
|
||||
// now that deps are installed we can import `@kbn/yarn-lock-validator`
|
||||
const { kibanaPackageJson } = External['@kbn/repo-info']();
|
||||
await validatePackageJson(kibanaPackageJson, log);
|
||||
}),
|
||||
validate
|
||||
? time('validate dependencies', async () => {
|
||||
// now that deps are installed we can import `@kbn/yarn-lock-validator`
|
||||
const { readYarnLock, validateDependencies } = External['@kbn/yarn-lock-validator']();
|
||||
await validateDependencies(log, await readYarnLock());
|
||||
})
|
||||
: undefined,
|
||||
vscodeConfig
|
||||
? time('update vscode config', async () => {
|
||||
// Update vscode settings
|
||||
await run('node', ['scripts/update_vscode_config']);
|
||||
|
||||
if (vscodeConfig) {
|
||||
await time('update vscode config', async () => {
|
||||
// Update vscode settings
|
||||
await run('node', ['scripts/update_vscode_config']);
|
||||
|
||||
log.success('vscode config updated');
|
||||
});
|
||||
}
|
||||
log.success('vscode config updated');
|
||||
})
|
||||
: undefined,
|
||||
]);
|
||||
},
|
||||
};
|
||||
|
|
|
|||
38
kbn_pm/src/commands/bootstrap/discovery.mjs
Normal file
38
kbn_pm/src/commands/bootstrap/discovery.mjs
Normal file
|
|
@ -0,0 +1,38 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0 and the Server Side Public License, v 1; you may not use this file except
|
||||
* in compliance with, at your election, the Elastic License 2.0 or the Server
|
||||
* Side Public License, v 1.
|
||||
*/
|
||||
|
||||
import { REPO_ROOT } from '../../lib/paths.mjs';
|
||||
|
||||
// we need to run these in order to generate the pkg map which is used by things
|
||||
// like `@kbn/babel-register`, so we have to import the JS files directory and can't
|
||||
// rely on `@kbn/babel-register`.
|
||||
|
||||
export async function packageDiscovery() {
|
||||
const { discoverBazelPackages } = await import(
|
||||
// eslint-disable-next-line @kbn/imports/uniform_imports
|
||||
'../../../../packages/kbn-bazel-packages/index.js'
|
||||
);
|
||||
|
||||
return await discoverBazelPackages(REPO_ROOT);
|
||||
}
|
||||
|
||||
export async function pluginDiscovery() {
|
||||
const { getPluginSearchPaths, simpleKibanaPlatformPluginDiscovery } = await import(
|
||||
// eslint-disable-next-line @kbn/imports/uniform_imports
|
||||
'../../../../packages/kbn-plugin-discovery/index.js'
|
||||
);
|
||||
|
||||
const searchPaths = getPluginSearchPaths({
|
||||
rootDir: REPO_ROOT,
|
||||
examples: true,
|
||||
oss: false,
|
||||
testPlugins: true,
|
||||
});
|
||||
|
||||
return simpleKibanaPlatformPluginDiscovery(searchPaths, []);
|
||||
}
|
||||
|
|
@ -10,47 +10,23 @@ import Path from 'path';
|
|||
import Fsp from 'fs/promises';
|
||||
|
||||
import { REPO_ROOT } from '../../lib/paths.mjs';
|
||||
import { convertPluginIdToPackageId } from './plugins.mjs';
|
||||
import { normalizePath } from './normalize_path.mjs';
|
||||
import External from '../../lib/external_packages.js';
|
||||
|
||||
/**
|
||||
* @param {import('@kbn/bazel-packages').BazelPackage[]} packages
|
||||
* @param {import('@kbn/plugin-discovery').KibanaPlatformPlugin[]} plugins
|
||||
*/
|
||||
export async function regenerateBaseTsconfig(packages, plugins) {
|
||||
export async function regenerateBaseTsconfig() {
|
||||
const pkgMap = External['@kbn/package-map']().readPackageMap();
|
||||
const tsconfigPath = Path.resolve(REPO_ROOT, 'tsconfig.base.json');
|
||||
const lines = (await Fsp.readFile(tsconfigPath, 'utf-8')).split('\n');
|
||||
|
||||
const packagesMap = packages
|
||||
.slice()
|
||||
.sort((a, b) => a.normalizedRepoRelativeDir.localeCompare(b.normalizedRepoRelativeDir))
|
||||
.flatMap((p) => {
|
||||
if (!p.pkg) {
|
||||
return [];
|
||||
}
|
||||
|
||||
const id = p.pkg.name;
|
||||
const path = p.normalizedRepoRelativeDir;
|
||||
return [` "${id}": ["${path}"],`, ` "${id}/*": ["${path}/*"],`];
|
||||
});
|
||||
|
||||
const pluginsMap = plugins
|
||||
.slice()
|
||||
.sort((a, b) => a.manifestPath.localeCompare(b.manifestPath))
|
||||
.flatMap((p) => {
|
||||
const id = convertPluginIdToPackageId(p.manifest.id);
|
||||
const path = normalizePath(Path.relative(REPO_ROOT, p.directory));
|
||||
return [` "${id}": ["${path}"],`, ` "${id}/*": ["${path}/*"],`];
|
||||
});
|
||||
|
||||
const start = lines.findIndex((l) => l.trim() === '// START AUTOMATED PACKAGE LISTING');
|
||||
const end = lines.findIndex((l) => l.trim() === '// END AUTOMATED PACKAGE LISTING');
|
||||
|
||||
const current = await Fsp.readFile(tsconfigPath, 'utf8');
|
||||
const updated = [
|
||||
...lines.slice(0, start + 1),
|
||||
...packagesMap,
|
||||
...pluginsMap,
|
||||
...Array.from(pkgMap.entries()).flatMap(([moduleId, repoRelPath]) => [
|
||||
` "${moduleId}": ["${repoRelPath}"],`,
|
||||
` "${moduleId}/*": ["${repoRelPath}/*"],`,
|
||||
]),
|
||||
...lines.slice(end),
|
||||
].join('\n');
|
||||
|
||||
|
|
|
|||
57
kbn_pm/src/commands/bootstrap/regenerate_package_map.mjs
Normal file
57
kbn_pm/src/commands/bootstrap/regenerate_package_map.mjs
Normal file
|
|
@ -0,0 +1,57 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0 and the Server Side Public License, v 1; you may not use this file except
|
||||
* in compliance with, at your election, the Elastic License 2.0 or the Server
|
||||
* Side Public License, v 1.
|
||||
*/
|
||||
|
||||
import Path from 'path';
|
||||
import Fs from 'fs';
|
||||
import Fsp from 'fs/promises';
|
||||
|
||||
import { convertPluginIdToPackageId } from '../../lib/plugins.mjs';
|
||||
import { normalizePath } from '../../lib/normalize_path.mjs';
|
||||
import { REPO_ROOT } from '../../lib/paths.mjs';
|
||||
|
||||
/**
|
||||
*
|
||||
* @param {import('@kbn/bazel-packages').BazelPackage[]} packages
|
||||
* @param {import('@kbn/plugin-discovery').KibanaPlatformPlugin[]} plugins
|
||||
* @param {import('@kbn/some-dev-log').SomeDevLog} log
|
||||
*/
|
||||
export async function regeneratePackageMap(packages, plugins, log) {
|
||||
// clean up old version of package map package
|
||||
Fs.rmSync(Path.resolve(REPO_ROOT, 'packages/kbn-synthetic-package-map'), {
|
||||
recursive: true,
|
||||
force: true,
|
||||
});
|
||||
|
||||
const path = Path.resolve(REPO_ROOT, 'packages/kbn-package-map/package-map.json');
|
||||
const existingContent = Fs.existsSync(path) ? await Fsp.readFile(path, 'utf8') : undefined;
|
||||
|
||||
/** @type {Array<[string, string]>} */
|
||||
const entries = [['@kbn/core', 'src/core']];
|
||||
|
||||
for (const pkg of packages) {
|
||||
entries.push([pkg.manifest.id, pkg.normalizedRepoRelativeDir]);
|
||||
}
|
||||
|
||||
for (const plugin of plugins) {
|
||||
entries.push([
|
||||
convertPluginIdToPackageId(plugin.manifest.id),
|
||||
normalizePath(Path.relative(REPO_ROOT, plugin.directory)),
|
||||
]);
|
||||
}
|
||||
|
||||
const content = JSON.stringify(
|
||||
entries.sort((a, b) => a[0].localeCompare(b[0])),
|
||||
null,
|
||||
2
|
||||
);
|
||||
|
||||
if (content !== existingContent) {
|
||||
await Fsp.writeFile(path, content);
|
||||
log.warning('updated package map, many caches may be invalidated');
|
||||
}
|
||||
}
|
||||
|
|
@ -1,34 +0,0 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0 and the Server Side Public License, v 1; you may not use this file except
|
||||
* in compliance with, at your election, the Elastic License 2.0 or the Server
|
||||
* Side Public License, v 1.
|
||||
*/
|
||||
|
||||
import Path from 'path';
|
||||
import Fsp from 'fs/promises';
|
||||
|
||||
import { normalizePath } from './normalize_path.mjs';
|
||||
import { REPO_ROOT } from '../../lib/paths.mjs';
|
||||
import { convertPluginIdToPackageId } from './plugins.mjs';
|
||||
|
||||
/**
|
||||
* @param {import('@kbn/plugin-discovery').KibanaPlatformPlugin[]} plugins
|
||||
*/
|
||||
export async function regenerateSyntheticPackageMap(plugins) {
|
||||
/** @type {Array<[string, string]>} */
|
||||
const entries = [['@kbn/core', 'src/core']];
|
||||
|
||||
for (const plugin of plugins) {
|
||||
entries.push([
|
||||
convertPluginIdToPackageId(plugin.manifest.id),
|
||||
normalizePath(Path.relative(REPO_ROOT, plugin.directory)),
|
||||
]);
|
||||
}
|
||||
|
||||
await Fsp.writeFile(
|
||||
Path.resolve(REPO_ROOT, 'packages/kbn-synthetic-package-map/synthetic-packages.json'),
|
||||
JSON.stringify(entries, null, 2)
|
||||
);
|
||||
}
|
||||
|
|
@ -10,9 +10,10 @@ import Path from 'path';
|
|||
import Fs from 'fs';
|
||||
|
||||
import { REPO_ROOT } from '../../lib/paths.mjs';
|
||||
import External from '../../lib/external_packages.js';
|
||||
|
||||
export async function sortPackageJson() {
|
||||
const { sortPackageJson } = await import('@kbn/sort-package-json');
|
||||
const { sortPackageJson } = External['@kbn/sort-package-json']();
|
||||
|
||||
const path = Path.resolve(REPO_ROOT, 'package.json');
|
||||
const json = Fs.readFileSync(path, 'utf8');
|
||||
|
|
|
|||
29
kbn_pm/src/commands/bootstrap/validate_package_json.mjs
Normal file
29
kbn_pm/src/commands/bootstrap/validate_package_json.mjs
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0 and the Server Side Public License, v 1; you may not use this file except
|
||||
* in compliance with, at your election, the Elastic License 2.0 or the Server
|
||||
* Side Public License, v 1.
|
||||
*/
|
||||
|
||||
import { createCliError } from '../../lib/cli_error.mjs';
|
||||
|
||||
/**
|
||||
* @param {import('@kbn/repo-info').KibanaPackageJson} pkgJson
|
||||
* @param {import('@kbn/some-dev-log').SomeDevLog} log
|
||||
*/
|
||||
export async function validatePackageJson(pkgJson, log) {
|
||||
const failures = false;
|
||||
|
||||
const typesInProd = Object.keys(pkgJson.dependencies).filter((id) => id.startsWith('@types/'));
|
||||
if (typesInProd.length) {
|
||||
const list = typesInProd.map((id) => ` - ${id}`).join('\n');
|
||||
log.error(
|
||||
`The following @types/* packages are listed in dependencies but should be in the devDependencies:\n${list}`
|
||||
);
|
||||
}
|
||||
|
||||
if (failures) {
|
||||
throw createCliError('failed to validate package.json, check for errors above');
|
||||
}
|
||||
}
|
||||
|
|
@ -1,10 +0,0 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0 and the Server Side Public License, v 1; you may not use this file except
|
||||
* in compliance with, at your election, the Elastic License 2.0 or the Server
|
||||
* Side Public License, v 1.
|
||||
*/
|
||||
|
||||
const { PROJECTS } = require('../../../src/dev/typescript/projects');
|
||||
module.exports = { PROJECTS };
|
||||
|
|
@ -10,6 +10,7 @@ import Path from 'path';
|
|||
|
||||
import { REPO_ROOT } from '../lib/paths.mjs';
|
||||
import { run, spawnStreaming } from '../lib/spawn.mjs';
|
||||
import External from '../lib/external_packages.js';
|
||||
|
||||
/** @type {import('../lib/command').Command} */
|
||||
export const command = {
|
||||
|
|
@ -39,7 +40,7 @@ export const command = {
|
|||
const exclude = args.getStringValues('exclude') ?? [];
|
||||
const include = args.getStringValues('include') ?? [];
|
||||
|
||||
const { discoverBazelPackages } = await import('@kbn/bazel-packages');
|
||||
const { discoverBazelPackages } = External['@kbn/bazel-packages']();
|
||||
const packages = await discoverBazelPackages(REPO_ROOT);
|
||||
for (const { manifest, pkg, normalizedRepoRelativeDir } of packages) {
|
||||
if (
|
||||
|
|
|
|||
|
|
@ -6,267 +6,8 @@
|
|||
* Side Public License, v 1.
|
||||
*/
|
||||
|
||||
import Fs from 'fs';
|
||||
import Path from 'path';
|
||||
|
||||
import { REPO_ROOT } from '../lib/paths.mjs';
|
||||
import { pluginDiscovery } from './bootstrap/plugins.mjs';
|
||||
|
||||
const RULE_DEPS = /([\s\n]deps\s*=\s*)((?:\w+(?: \+ )?)?(?:\[[^\]]*\])?)(\s*,|\s*\))/;
|
||||
|
||||
/**
|
||||
* @param {string} text
|
||||
* @param {number} index
|
||||
*/
|
||||
function findStartOfLine(text, index) {
|
||||
let cursor = index;
|
||||
while (cursor > 0) {
|
||||
if (text[cursor - 1] === '\n') {
|
||||
return cursor;
|
||||
}
|
||||
cursor -= 1;
|
||||
}
|
||||
|
||||
return cursor;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {string} starlark
|
||||
* @param {string} name
|
||||
*/
|
||||
function findBazelRule(starlark, name) {
|
||||
const match = starlark.match(new RegExp(`name\\s*=\\s*${name}`));
|
||||
if (typeof match?.index !== 'number') {
|
||||
throw new Error(`unable to find rule named [${name}]`);
|
||||
}
|
||||
|
||||
const openParen = starlark.slice(0, match.index).lastIndexOf('(');
|
||||
if (openParen === -1) {
|
||||
throw new Error(`unable to find opening paren for rule [${name}] [index=${match.index}]`);
|
||||
}
|
||||
|
||||
const start = findStartOfLine(starlark, openParen);
|
||||
const end = starlark.indexOf(')', start);
|
||||
if (end === -1) {
|
||||
throw new Error(`unable to find closing parent for rule [${name}] [start=${start}]`);
|
||||
}
|
||||
|
||||
const type = starlark.slice(start, starlark.indexOf('(', start)).trim();
|
||||
|
||||
// add 1 so that the "end" chunk starts after the closing )
|
||||
return { start, end: end + 1, type };
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {string} starlark
|
||||
* @param {string} name
|
||||
*/
|
||||
function removeBazelRule(starlark, name) {
|
||||
const pos = findBazelRule(starlark, name);
|
||||
|
||||
let end = pos.end;
|
||||
|
||||
// slurp up all the newlines directly after the closing )
|
||||
while (starlark[end] === '\n') {
|
||||
end += 1;
|
||||
}
|
||||
|
||||
return starlark.slice(0, pos.start) + starlark.slice(end);
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {string} starlark
|
||||
* @param {string} dep
|
||||
* @returns
|
||||
*/
|
||||
function addDep(starlark, dep) {
|
||||
const depsMatch = starlark.match(RULE_DEPS);
|
||||
|
||||
if (typeof depsMatch?.index !== 'number') {
|
||||
return starlark.replace(/,?[\s\n]*\)[\s\n]*$/, '') + `,\n deps = [${dep}],\n)`;
|
||||
}
|
||||
|
||||
const [, head, value, tail] = depsMatch;
|
||||
|
||||
return (
|
||||
starlark.slice(0, depsMatch.index) +
|
||||
head +
|
||||
(() => {
|
||||
const multiline = value.includes('\n');
|
||||
const existingArray = value.indexOf(']');
|
||||
if (existingArray === -1) {
|
||||
return value + ` + [${dep}]`;
|
||||
}
|
||||
|
||||
const valHead = value.slice(0, existingArray).replace(/,?\s*$/, ',');
|
||||
const valTail = value.slice(existingArray);
|
||||
|
||||
return `${valHead}${multiline ? '\n ' : ' '}${dep}${multiline ? ',\n' : ''}${valTail}`;
|
||||
})() +
|
||||
tail +
|
||||
starlark.slice(depsMatch.index + depsMatch[0].length)
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {string} starlark
|
||||
* @param {string} name
|
||||
* @param {string} newName
|
||||
* @param {(rule: string) => string} mod
|
||||
*/
|
||||
function duplicateRule(starlark, name, newName, mod) {
|
||||
const origPos = findBazelRule(starlark, name);
|
||||
|
||||
const orig = starlark.slice(origPos.start, origPos.end);
|
||||
|
||||
const withName = orig.replace(
|
||||
/^(\s*)name\s*=\s*.*$/m,
|
||||
(match, head) => `${head}name = ${newName}${match.endsWith(',') ? ',' : ''}`
|
||||
);
|
||||
|
||||
return starlark.slice(0, origPos.end) + `\n\n${mod(withName)}` + starlark.slice(origPos.end);
|
||||
}
|
||||
|
||||
/** @type {import('../lib/command').Command} */
|
||||
export const command = {
|
||||
name: '_x',
|
||||
async run({ log }) {
|
||||
const updates = { pkgJson: 0, buildBazel: 0, tsconfig: 0, tsconfigRefs: 0 };
|
||||
|
||||
await import('../../../src/setup_node_env/index' + '.js');
|
||||
const { PROJECTS } = await import('./projects' + '.js');
|
||||
const { discoverBazelPackages } = await import('@kbn/bazel-packages');
|
||||
const pkgs = await discoverBazelPackages(REPO_ROOT);
|
||||
const plugins = await pluginDiscovery();
|
||||
|
||||
// update package.json files to point to their target_types dir
|
||||
const relTypes = './target_types/index.d.ts';
|
||||
for (const pkg of pkgs) {
|
||||
if (!pkg.hasBuildTypesRule()) {
|
||||
log.warning(`not defining "types" for ${pkg.manifest.id} because it doesn't build types`);
|
||||
continue;
|
||||
}
|
||||
|
||||
const dir = Path.resolve(REPO_ROOT, pkg.normalizedRepoRelativeDir);
|
||||
const pkgJsonPath = Path.resolve(dir, 'package.json');
|
||||
|
||||
const pkgJson = Fs.readFileSync(pkgJsonPath, 'utf8');
|
||||
const parsed = JSON.parse(pkgJson);
|
||||
|
||||
if (parsed.types === relTypes) {
|
||||
continue;
|
||||
}
|
||||
|
||||
Fs.writeFileSync(
|
||||
pkgJsonPath,
|
||||
JSON.stringify(
|
||||
{
|
||||
...parsed,
|
||||
types: relTypes,
|
||||
},
|
||||
null,
|
||||
2
|
||||
) + (pkgJson.endsWith('\n') ? '\n' : '')
|
||||
);
|
||||
|
||||
updates.pkgJson += 1;
|
||||
}
|
||||
log.success(`updated ${updates.pkgJson} package.json files`);
|
||||
|
||||
// update BUILD.bazel files to not rely on type_summarizer
|
||||
for (const pkg of pkgs) {
|
||||
if (!pkg.hasBuildTypesRule()) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const starlark = pkg.buildBazelContent;
|
||||
if (typeof starlark !== 'string') {
|
||||
throw new Error('missing buildBazelContent');
|
||||
}
|
||||
|
||||
const npmTypes = findBazelRule(starlark, '"npm_module_types"');
|
||||
|
||||
if (npmTypes.type === 'alias') {
|
||||
log.info(`ignoring npm_module_types rule which is an alias in ${pkg.manifest.id}`);
|
||||
continue;
|
||||
}
|
||||
|
||||
// remove rules for old npm_module_types
|
||||
const withoutOldTypes = removeBazelRule(starlark, '"npm_module_types"');
|
||||
|
||||
// duplicate js_library rule and name npm_module_types rule which adds the ':tsc_types' dep
|
||||
const withTypesJsLib = duplicateRule(
|
||||
withoutOldTypes,
|
||||
'PKG_DIRNAME',
|
||||
'"npm_module_types"',
|
||||
(newRule) => addDep(newRule, '":tsc_types"')
|
||||
);
|
||||
|
||||
const withBuildTypesWrapper =
|
||||
removeBazelRule(withTypesJsLib, '"build_types"').trimEnd() +
|
||||
`
|
||||
|
||||
pkg_npm(
|
||||
name = "build_types",
|
||||
deps = [":npm_module_types"],
|
||||
visibility = ["//visibility:public"],
|
||||
)
|
||||
`;
|
||||
|
||||
Fs.writeFileSync(
|
||||
Path.resolve(REPO_ROOT, pkg.normalizedRepoRelativeDir, 'BUILD.bazel'),
|
||||
withBuildTypesWrapper
|
||||
);
|
||||
|
||||
updates.buildBazel += 1;
|
||||
}
|
||||
log.success(`updated ${updates.buildBazel} BUILD.bazel files`);
|
||||
|
||||
// stop enabling declaration source maps in tsconfig
|
||||
for (const pkg of [...pkgs, ...plugins]) {
|
||||
const dir =
|
||||
'normalizedRepoRelativeDir' in pkg
|
||||
? Path.resolve(REPO_ROOT, pkg.normalizedRepoRelativeDir)
|
||||
: pkg.directory;
|
||||
|
||||
let changed;
|
||||
|
||||
const tsconfigPath = Path.resolve(dir, 'tsconfig.json');
|
||||
if (Fs.existsSync(tsconfigPath)) {
|
||||
const current = Fs.readFileSync(tsconfigPath, 'utf8');
|
||||
const next = current.replace(/\n\s*"declarationMap"\s*:.+\n/m, '\n');
|
||||
|
||||
if (current !== next) {
|
||||
changed = true;
|
||||
Fs.writeFileSync(tsconfigPath, next);
|
||||
}
|
||||
}
|
||||
|
||||
const buildBazelPath = Path.resolve(dir, 'BUILD.bazel');
|
||||
if (Fs.existsSync(buildBazelPath)) {
|
||||
const current = Fs.readFileSync(buildBazelPath, 'utf8');
|
||||
const next = current.replace(/\n.*\bdeclaration_map\b.*\n/, '\n');
|
||||
if (current !== next) {
|
||||
changed = true;
|
||||
Fs.writeFileSync(buildBazelPath, next);
|
||||
}
|
||||
}
|
||||
|
||||
if (changed) {
|
||||
updates.tsconfig += 1;
|
||||
}
|
||||
}
|
||||
log.success(`dropped declarationMap from ${updates.tsconfig} tsconfig.json files`);
|
||||
|
||||
// rename "references" in plugin tsconfig.json files to "kbn_references"
|
||||
for (const project of PROJECTS) {
|
||||
const tsconfigJson = Fs.readFileSync(project.tsConfigPath, 'utf8');
|
||||
const updated = tsconfigJson.replace('"references"', '"kbn_references"');
|
||||
if (updated !== tsconfigJson) {
|
||||
Fs.writeFileSync(project.tsConfigPath, updated);
|
||||
updates.tsconfigRefs += 1;
|
||||
}
|
||||
}
|
||||
log.success(`updated tsconfig references key in ${updates.tsconfigRefs} tsconfig.json files`);
|
||||
},
|
||||
async run() {},
|
||||
};
|
||||
|
|
|
|||
|
|
@ -18,6 +18,12 @@ import { indent } from './indent.mjs';
|
|||
|
||||
const BAZEL_RUNNER_SRC = '../../../packages/kbn-bazel-runner/index.js';
|
||||
|
||||
const BAZEL_TARGETS = [
|
||||
'//packages/kbn-ui-shared-deps-npm:shared_built_assets',
|
||||
'//packages/kbn-ui-shared-deps-src:shared_built_assets',
|
||||
'//packages/kbn-monaco:target_workers',
|
||||
];
|
||||
|
||||
async function getBazelRunner() {
|
||||
/* eslint-disable no-unsanitized/method */
|
||||
/** @type {import('@kbn/bazel-runner')} */
|
||||
|
|
@ -83,7 +89,7 @@ export async function watch(log, opts = undefined) {
|
|||
// `.bazel_fix_commands.json` but its not needed at the moment
|
||||
'--run_output=false',
|
||||
'build',
|
||||
'//packages:build',
|
||||
...BAZEL_TARGETS,
|
||||
'--show_result=1',
|
||||
...(opts?.offline ? ['--config=offline'] : []),
|
||||
];
|
||||
|
|
@ -158,13 +164,13 @@ export async function installYarnDeps(log, opts = undefined) {
|
|||
* @param {import('./log.mjs').Log} log
|
||||
* @param {{ offline?: boolean, quiet?: boolean } | undefined} opts
|
||||
*/
|
||||
export async function buildPackages(log, opts = undefined) {
|
||||
await runBazel(log, ['build', '//packages:build', '--show_result=1'], {
|
||||
export async function buildWebpackBundles(log, opts = undefined) {
|
||||
await runBazel(log, ['build', ...BAZEL_TARGETS, '--show_result=1'], {
|
||||
offline: opts?.offline,
|
||||
quiet: opts?.quiet,
|
||||
});
|
||||
|
||||
log.success('packages built');
|
||||
log.success('shared bundles built');
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
|
|||
58
kbn_pm/src/lib/external_packages.js
Normal file
58
kbn_pm/src/lib/external_packages.js
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0 and the Server Side Public License, v 1; you may not use this file except
|
||||
* in compliance with, at your election, the Elastic License 2.0 or the Server
|
||||
* Side Public License, v 1.
|
||||
*/
|
||||
module.exports = {
|
||||
['@kbn/bazel-packages']() {
|
||||
require('@kbn/babel-register').install();
|
||||
return require('@kbn/bazel-packages');
|
||||
},
|
||||
|
||||
['@kbn/ci-stats-reporter']() {
|
||||
require('@kbn/babel-register').install();
|
||||
return require('@kbn/ci-stats-reporter');
|
||||
},
|
||||
|
||||
['@kbn/yarn-lock-validator']() {
|
||||
require('@kbn/babel-register').install();
|
||||
return require('@kbn/yarn-lock-validator');
|
||||
},
|
||||
|
||||
['@kbn/sort-package-json']() {
|
||||
require('@kbn/babel-register').install();
|
||||
return require('@kbn/sort-package-json');
|
||||
},
|
||||
|
||||
['@kbn/package-map']() {
|
||||
require('@kbn/babel-register').install();
|
||||
return require('@kbn/package-map');
|
||||
},
|
||||
|
||||
['@kbn/get-repo-files']() {
|
||||
require('@kbn/babel-register').install();
|
||||
return require('@kbn/get-repo-files');
|
||||
},
|
||||
|
||||
['@kbn/repo-info']() {
|
||||
require('@kbn/babel-register').install();
|
||||
return require('@kbn/repo-info');
|
||||
},
|
||||
|
||||
['@kbn/ts-projects']() {
|
||||
require('@kbn/babel-register').install();
|
||||
return require('@kbn/ts-projects');
|
||||
},
|
||||
|
||||
/**
|
||||
* @param {string} absPath
|
||||
* @returns {unknown}
|
||||
*/
|
||||
reqAbs(absPath) {
|
||||
require('@kbn/babel-register').install();
|
||||
// eslint-disable-next-line import/no-dynamic-require
|
||||
return require(absPath);
|
||||
},
|
||||
};
|
||||
|
|
@ -10,21 +10,20 @@ import Path from 'path';
|
|||
import Fs from 'fs';
|
||||
|
||||
import { REPO_ROOT } from './paths.mjs';
|
||||
import External from './external_packages.js';
|
||||
|
||||
/**
|
||||
* Attempt to load the synthetic package map, if bootstrap hasn't run successfully
|
||||
* Attempt to load the package map, if bootstrap hasn't run successfully
|
||||
* this might fail.
|
||||
* @param {import('@kbn/some-dev-log').SomeDevLog} log
|
||||
* @returns {Promise<import('@kbn/synthetic-package-map').PackageMap>}
|
||||
* @returns {Promise<import('@kbn/package-map').PackageMap>}
|
||||
*/
|
||||
async function tryToGetSyntheticPackageMap(log) {
|
||||
async function tryToGetPackageMap(log) {
|
||||
try {
|
||||
const { readPackageMap } = await import('@kbn/synthetic-package-map');
|
||||
const { readPackageMap } = External['@kbn/package-map']();
|
||||
return readPackageMap();
|
||||
} catch (error) {
|
||||
log.warning(
|
||||
'unable to load synthetic package map, unable to clean target directories in synthetic packages'
|
||||
);
|
||||
log.warning('unable to load package map, unable to clean target directories in packages');
|
||||
return new Map();
|
||||
}
|
||||
}
|
||||
|
|
@ -67,7 +66,7 @@ export function readCleanPatterns(packageDir) {
|
|||
* @returns {Promise<string[]>}
|
||||
*/
|
||||
export async function findPluginCleanPaths(log) {
|
||||
const packageMap = await tryToGetSyntheticPackageMap(log);
|
||||
const packageMap = await tryToGetPackageMap(log);
|
||||
return [...packageMap.values()].flatMap((repoRelativePath) => {
|
||||
const pkgDir = Path.resolve(REPO_ROOT, repoRelativePath);
|
||||
return [Path.resolve(pkgDir, 'target'), ...readCleanPatterns(pkgDir)];
|
||||
|
|
|
|||
|
|
@ -6,14 +6,8 @@
|
|||
* Side Public License, v 1.
|
||||
*/
|
||||
|
||||
import { REPO_ROOT } from '../../lib/paths.mjs';
|
||||
|
||||
/** @type {string} */
|
||||
const PLUGIN_DISCOVERY_SRC = '../../../../packages/kbn-plugin-discovery/index.js';
|
||||
|
||||
/**
|
||||
* @param {string} pluginId
|
||||
* @returns {string}
|
||||
*/
|
||||
export function convertPluginIdToPackageId(pluginId) {
|
||||
if (pluginId === 'core') {
|
||||
|
|
@ -28,24 +22,3 @@ export function convertPluginIdToPackageId(pluginId) {
|
|||
.replace(/-\w(-\w)+-/g, (match) => `-${match.split('-').join('')}-`)
|
||||
.replace(/-plugin-plugin$/, '-plugin');
|
||||
}
|
||||
|
||||
/**
|
||||
* @returns {Promise<import('@kbn/plugin-discovery').KibanaPlatformPlugin[]>}
|
||||
*/
|
||||
export async function pluginDiscovery() {
|
||||
/* eslint-disable no-unsanitized/method */
|
||||
/** @type {import('@kbn/plugin-discovery')} */
|
||||
const { getPluginSearchPaths, simpleKibanaPlatformPluginDiscovery } = await import(
|
||||
PLUGIN_DISCOVERY_SRC
|
||||
);
|
||||
/* eslint-enable no-unsanitized/method */
|
||||
|
||||
const searchPaths = getPluginSearchPaths({
|
||||
rootDir: REPO_ROOT,
|
||||
examples: true,
|
||||
oss: false,
|
||||
testPlugins: true,
|
||||
});
|
||||
|
||||
return simpleKibanaPlatformPluginDiscovery(searchPaths, []);
|
||||
}
|
||||
|
|
@ -1,14 +1,28 @@
|
|||
{
|
||||
"extends": "../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"outDir": "target",
|
||||
"outDir": "target/types",
|
||||
"checkJs": true,
|
||||
"target": "ES2022",
|
||||
"module": "ESNext"
|
||||
},
|
||||
"include": [
|
||||
"src/**/*.mjs",
|
||||
"src/**/*.js",
|
||||
"src/**/*.ts",
|
||||
],
|
||||
"exclude": []
|
||||
"exclude": [
|
||||
"target/**/*",
|
||||
],
|
||||
"kbn_references": [
|
||||
"@kbn/babel-register",
|
||||
"@kbn/bazel-packages",
|
||||
"@kbn/repo-info",
|
||||
"@kbn/yarn-lock-validator",
|
||||
"@kbn/get-repo-files",
|
||||
"@kbn/sort-package-json",
|
||||
{ "path": "../src/dev/tsconfig.json" },
|
||||
"@kbn/ci-stats-reporter",
|
||||
"@kbn/package-map",
|
||||
"@kbn/ts-projects"
|
||||
]
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue