Switch to Yarn (#15485)

* switch to yarn

* cleanup misc references to npm

* [yarn] loosen dependency ranges so yarn will merge more deps

* fix linting error now that moment uses ESM

* [licenses] font-awesome changed the format of its license id

* Use local yarn

* Misc fixes

* eslintignore built yarn file

* Remove mkdir which doesn't do what it should do

* Check build without upgrading lots of versions

* Fix license check

* too many moments

* Better description

* Review fixes

* Lock to angular@1.6.5

* More specific version locks

* Revert "More specific version locks"

This reverts commit 11ef81102e.

* Revert "Lock to angular@1.6.5"

This reverts commit 3ade68c14c.

* rm yarn.lock; yarn

* Forcing a specific version of React, Angular, Moment

* Using vendored version of yarn in ci

* Use --frozen-lockfile

* fixes

.eslintignore

@@ -8,3 +8,4 @@
 /src/ui/public/utils/decode_geo_hash.js
 /src/core_plugins/timelion/public/webpackShims/jquery.flot.*
 /ui_framework/doc_site/build
+/tasks/vendor

.github/PULL_REQUEST_TEMPLATE.md

@@ -8,6 +8,6 @@ attention.
 - Have you signed the [contributor license agreement](https://www.elastic.co/contributor-agreement)?
 - Have you followed the [contributor guidelines](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md)?
 - If submitting code, have you included unit tests that cover the changes?
-- If submitting code, have you tested and built your code locally prior to submission with `npm test && npm run build`?
+- If submitting code, have you tested and built your code locally prior to submission with `yarn test && yarn build`?
 - If submitting code, is your pull request against master? Unless there is a good reason otherwise, we prefer pull requests against master and will backport as needed.
--->
+-->

.gitignore

@@ -35,7 +35,6 @@ selenium
 *.out
 ui_framework/doc_site/build
 !ui_framework/doc_site/build/index.html
-yarn.lock
 package-lock.json
 .yo-rc.json
 /.vscode

.npmrc

@@ -1 +0,0 @@
-save-exact=true

CONTRIBUTING.md

@@ -158,16 +158,18 @@ Install the version of node.js listed in the `.node-version` file _(this can be
 nvm install "$(cat .node-version)"
 ```
 
-Install `npm` dependencies
+Install the latest version of [yarn](https://yarnpkg.com).
+
+Install dependencies
 
 ```bash
-npm install
+yarn
 ```
 
 Start elasticsearch.
 
 ```bash
-npm run elasticsearch
+yarn elasticsearch
 ```
 
 > You'll need to have a `java` binary in `PATH` or set `JAVA_HOME`.
@@ -182,12 +184,12 @@ node scripts/makelogs
 
 Start the development server.
   ```bash
-  npm start
+  yarn start
   ```
 
 > On Windows, you'll need you use Git Bash, Cygwin, or a similar shell that exposes the `sh` command.  And to successfully build you'll need Cygwin optional packages zip, tar, and shasum.
 
-Now you can point your web browser to https://localhost:5601 and start using Kibana! When running `npm start`, Kibana will also log that it is listening on port 5603 due to the base path proxy, but you should still access Kibana on port 5601.
+Now you can point your web browser to https://localhost:5601 and start using Kibana! When running `yarn start`, Kibana will also log that it is listening on port 5603 due to the base path proxy, but you should still access Kibana on port 5601.
 
 #### Customizing `config/kibana.dev.yml`
 
@@ -203,7 +205,7 @@ In development mode, Kibana runs a customized version of [Webpack](http://webpac
 
 #### Setting Up SSL
 
-Kibana includes a self-signed certificate that can be used for development purposes: `npm start -- --ssl`.
+Kibana includes a self-signed certificate that can be used for development purposes: `yarn start --ssl`.
 
 ### Linting
 
@@ -230,23 +232,23 @@ Before running the tests you will need to install the projects dependencies as d
 Once that's done, just run:
 
 ```bash
-npm run test && npm run build -- --skip-os-packages
+yarn test && yarn build --skip-os-packages
 ```
 
 ### Debugging Unit Tests
 
-The standard `npm run test` task runs several sub tasks and can take several minutes to complete, making debugging failures pretty painful. In order to ease the pain specialized tasks provide alternate methods for running the tests.
+The standard `yarn test` task runs several sub tasks and can take several minutes to complete, making debugging failures pretty painful. In order to ease the pain specialized tasks provide alternate methods for running the tests.
 
-To execute both server and browser tests, but skip linting, use `npm run test:quick`.
+To execute both server and browser tests, but skip linting, use `yarn test:quick`.
 
 ```bash
-npm run test:quick
+yarn test:quick
 ```
 
-Use `npm run test:server` when you want to run only the server tests.
+Use `yarn test:server` when you want to run only the server tests.
 
 ```bash
-npm run test:server
+yarn test:server
 ```
 
 When you'd like to execute individual server-side test files, you can use the command below. Note that this command takes care of configuring Mocha with Babel compilation for you, and you'll be better off avoiding a globally installed `mocha` package. This command is great for development and for quickly identifying bugs.
@@ -261,16 +263,16 @@ You could also add the `--debug` option so that `node` is run using the `--debug
 node scripts/mocha --debug <file>
 ```
 
-With `npm run test:browser`, you can run only the browser tests. Coverage reports are available for browser tests by running `npm run test:coverage`. You can find the results under the `coverage/` directory that will be created upon completion.
+With `yarn test:browser`, you can run only the browser tests. Coverage reports are available for browser tests by running `yarn test:coverage`. You can find the results under the `coverage/` directory that will be created upon completion.
 
 ```bash
-npm run test:browser
+yarn test:browser
 ```
 
-Using `npm run test:dev` initializes an environment for debugging the browser tests. Includes an dedicated instance of the kibana server for building the test bundle, and a karma server. When running this task the build is optimized for the first time and then a karma-owned instance of the browser is opened. Click the "debug" button to open a new tab that executes the unit tests.
+Using `yarn test:dev` initializes an environment for debugging the browser tests. Includes an dedicated instance of the kibana server for building the test bundle, and a karma server. When running this task the build is optimized for the first time and then a karma-owned instance of the browser is opened. Click the "debug" button to open a new tab that executes the unit tests.
 
 ```bash
-npm run test:dev
+yarn test:dev
 ```
 
 In the screenshot below, you'll notice the URL is `localhost:9876/debug.html`. You can append a `grep` query parameter to this URL and set it to a string value which will be used to exclude tests which don't match. For example, if you changed the URL to `localhost:9876/debug.html?query=my test` and then refreshed the browser, you'd only see tests run which contain "my test" in the test description.
@@ -285,8 +287,8 @@ This should work super if you're using the [Kibana plugin generator](https://git
 To run the tests for just your particular plugin run the following command from your plugin:
 
 ```bash
-npm run test:server
-npm run test:browser -- --dev # remove the --dev flag to run them once and close
+yarn test:server
+yarn test:browser --dev # remove the --dev flag to run them once and close
 ```
 
 ### Cross-browser Compatibility
@@ -300,7 +302,7 @@ npm run test:browser -- --dev # remove the --dev flag to run them once and close
 * Open VMWare and go to Window > Virtual Machine Library. Unzip the virtual machine and drag the .vmx file into your Virtual Machine Library.
 * Right-click on the virtual machine you just added to your library and select "Snapshots...", and then click the "Take" button in the modal that opens. You can roll back to this snapshot when the VM expires in 90 days.
 * In System Preferences > Sharing, change your computer name to be something simple, e.g. "computer".
-* Run Kibana with `npm start -- --host=computer.local` (substituting your computer name).
+* Run Kibana with `yarn start --host=computer.local` (substituting your computer name).
 * Now you can run your VM, open the browser, and navigate to `http://computer.local:5601` to test Kibana.
 
 #### Running Browser Automation Tests
@@ -315,13 +317,13 @@ Packages are built using fpm, dpkg, and rpm.  Package building has only been tes
 ```bash
 apt-get install ruby-dev rpm
 gem install fpm -v 1.5.0
-npm run build -- --skip-archives
+yarn build --skip-archives
 ```
 
 To specify a package to build you can add `rpm` or `deb` as an argument.
 
 ```bash
-npm run build -- --rpm
+yarn build --rpm
 ```
 
 Distributable packages can be found in `target/` after the build completes.
@@ -381,4 +383,4 @@ Remember, someone is blocked by a pull awaiting review, make it count. Be thorou
 1. **Hand it off** If you're the first reviewer and everything looks good but the changes are more than a few lines, hand the pull to someone else to take a second look. Again, try to find the right person to assign it to.
 1. **Merge the code** When everything looks good, put in a `LGTM` (looks good to me) comment. Merge into the target branch. Check the labels on the pull to see if backporting is required, and perform the backport if so.
 
-Thank you so much for reading our guidelines! :tada:
+Thank you so much for reading our guidelines! :tada:

docs/development/core/development-basepath.asciidoc

@@ -79,9 +79,9 @@ To accomplish this the `serve` task does a few things:
 
 This proxy can sometimes have unintended side effects in development, so when
 needed you can opt out by passing the `--no-base-path` flag to the `serve` task
-or `npm start`.
+or `yarn start`.
 
 ["source","shell"]
 -----------
-npm start -- --no-base-path
------------
+yarn start --no-base-path
+-----------

docs/development/core/development-dependencies.asciidoc

@@ -10,14 +10,14 @@ Before you can use an external library with Kibana you have to install it. You
 do that using...
 
 [float]
-==== npm (preferred method)
+==== yarn (preferred method)
 
 Once you've http://npmsearch.com[found] a dependency you want to add, you can
 install it like so:
 
 ["source","shell"]
 -----------
-npm install --save some-neat-library
+yarn add some-neat-library
 -----------
 
 At the top of a javascript file, just import the library using it's name:
@@ -28,13 +28,13 @@ import someNeatLibrary from 'some-neat-library';
 -----------
 
 Just like working in node.js, front-end code can require node modules installed
-by npm without any additional configuration.
+by yarn without any additional configuration.
 
 [float]
 ==== webpackShims
 
 When a library you want to use does use es6 or common.js modules but is not
-available on npm, you can copy the source of the library into a webpackShim.
+available with yarn, you can copy the source of the library into a webpackShim.
 
 ["source","shell"]
 -----------
@@ -97,7 +97,7 @@ module.exports = window.angular;
 What this shim does is fairly simple if you go line by line:
 
 . makes sure that jQuery is loaded before angular (which actually runs the shim above)
-. load the angular.js file from the npm installation
+. load the angular.js file from the node_modules directory
 . load the angular-elastic plugin, a plugin we want to always be included whenever we import angular
 . use the `ui/modules` module to add the module exported by angular-elastic as a dependency to the `kibana` angular module
 . finally, export the window.angular variable. This means that writing `import angular from 'angular';` will properly set the angular variable to the angular library, rather than undefined which is the default behavior.

docs/development/core/development-functional-tests.asciidoc

@@ -12,20 +12,20 @@ There are three ways to run the tests depending on your goals:
 
 1. Easiest option:
 ** Description: Starts up Kibana & Elasticsearch servers, followed by running tests. This is much slower when running the tests multiple times because slow startup time for the servers. Recommended for single-runs.
-** `npm run test:ui`
+** `yarn test:ui`
 *** does everything in a single command, including running Elasticsearch and Kibana locally
 *** tears down everything after the tests run
 *** exit code reports success/failure of the tests
 
 2. Best for development:
 ** Description: Two commands, run in separate terminals, separate the components that are long-running and slow from those that are ephemeral and fast. Tests can be re-run much faster, and this still runs Elasticsearch & Kibana locally.
-** `npm run test:ui:server`
+** `yarn test:ui:server`
 *** starts Elasticsearch and Kibana servers
 *** slow to start
 *** can be reused for multiple executions of the tests, thereby saving some time when re-running tests
 *** automatically restarts the Kibana server when relevant changes are detected
 ** `node scripts/functional_test_runner`
-*** runs the tests against Kibana & Elasticsearch servers that were started `npm run test:ui:server`
+*** runs the tests against Kibana & Elasticsearch servers that were started `yarn test:ui:server`
 *** exit code reports success or failure of the tests
 
 3. Custom option:
@@ -405,4 +405,4 @@ const log = getService(‘log’);
 
 // log.debug only writes when using the `--debug` or `--verbose` flag.
 log.debug(‘done clicking menu’);
------------
+-----------

package.json

@@ -298,8 +298,13 @@
     "yeoman-generator": "1.1.1",
     "yo": "2.0.0"
   },
+  "resolutions": {
+    "angular": "1.6.5",
+    "moment": "2.13.0",
+    "react": "16.0.0"
+  },
   "engines": {
     "node": "6.12.2",
-    "npm": "3.10.10"
+    "yarn": "^1.3.2"
   }
 }

src/dev/ci_setup/setup.sh

@@ -10,7 +10,7 @@ cacheDir="${CACHE_DIR:-"/tmp/kibana"}"
 ### check that we seem to be in a kibana project
 ###
 if [ -f "$dir/package.json" ] && [ -f "$dir/.node-version" ]; then
-  echo "Setting up node.js and npm in $dir"
+  echo "Setting up node.js and yarn in $dir"
 else
   echo "src/dev/ci_setup/setup.sh must be run within a kibana repo"
   exit 1
@@ -41,13 +41,33 @@ fi
 
 
 ###
-### "install" node by extending the path with it's bin directory
+### "install" node into this shell
 ###
 export PATH="$nodeDir/bin:$PATH"
+hash -r
+
+###
+### setting up yarn
+###
+yarnVersion="1.3.2"
+yarnDir="$cacheDir/yarn/$yarnVersion"
+
+echo " -- using vendored version of yarn"
+mkdir -p "$yarnDir/bin"
+cp "$dir/tasks/vendor/yarn-1.3.2-with-ignore-fix.js" "$yarnDir/bin/yarn"
+
+###
+### "install" yarn into this shell
+###
+export PATH="$yarnDir/bin:$PATH"
+yarnGlobalDir="$(yarn global bin)"
+export PATH="$PATH:$yarnGlobalDir"
+hash -r
 
 
 ###
 ### install dependencies
 ###
 echo " -- installing node.js dependencies"
-npm install --cache "$cacheDir/npm"
+yarn config set cache-folder "$cacheDir/yarn"
+yarn --frozen-lockfile

src/ui/public/time_buckets/calc_auto_interval.js

@@ -1,4 +1,5 @@
-import moment, { duration as d } from 'moment';
+import moment from 'moment';
+const { duration: d } = moment;
 
 export function TimeBucketsCalcAutoIntervalProvider() {
   // these are the rounding rules used by roundInterval()

tasks/build/index.js

@@ -15,7 +15,7 @@ module.exports = function (grunt) {
       '_build:packageJson',
       '_build:readme',
       '_build:babelCache',
-      '_build:installNpmDeps',
+      '_build:installDependencies',
       '_build:notice',
       '_build:removePkgJsonDeps',
       'clean:testsFromModules',

tasks/build/install_dependencies.js

@@ -0,0 +1,13 @@
+import { exec } from 'child_process';
+module.exports = function (grunt) {
+  grunt.registerTask('_build:installDependencies', function () {
+    // We rely on a local version of Yarn that contains the bugfix from
+    // https://github.com/yarnpkg/yarn/pull/5059. Once this fix is merged
+    // and released we can use Yarn directly in the build.
+    const yarn = require.resolve('../vendor/yarn-1.3.2-with-ignore-fix.js');
+
+    exec(`${yarn} --production --ignore-optional --frozen-lockfile`, {
+      cwd: grunt.config.process('<%= root %>/build/kibana')
+    }, this.async());
+  });
+};

tasks/build/install_npm_deps.js

@@ -1,12 +0,0 @@
-import { exec } from 'child_process';
-module.exports = function (grunt) {
-  grunt.registerTask('_build:installNpmDeps', function () {
-    grunt.file.mkdir('build/kibana/node_modules');
-
-    exec('npm install  --production --no-optional', {
-      cwd: grunt.config.process('<%= root %>/build/kibana')
-    }, this.async());
-  });
-};
-
-

tasks/build/remove_pkg_json_deps.js

@@ -8,5 +8,7 @@ module.exports = function (grunt) {
       'build/kibana/package.json',
       JSON.stringify(pkg, null, '  ')
     );
+
+    grunt.file.delete('build/kibana/yarn.lock');
   });
 };

tasks/config/copy.js

@@ -3,6 +3,7 @@ module.exports = function () {
     devSource: {
       options: { mode: true },
       src: [
+        'yarn.lock',
         'src/**',
         '!src/**/__tests__/**',
         '!src/test_utils/**',

tasks/config/karma.js

@@ -102,11 +102,11 @@ module.exports = function (grunt) {
    *  (&shard_num=Y), are added to the testing bundle url and read by the
    *  test_harness/setup_test_sharding[1] module. This allows us to use a
    *  different number of shards in different scenarios (ie. running
-   *  `npm run test:browser` runs the tests in a single shard, effectively
+   *  `yarn test:browser` runs the tests in a single shard, effectively
    *  disabling sharding)
    *
    *  These same parameters can also be defined in the URL/query string of the
-   *  karma debug page (started when you run `npm run test:dev`).
+   *  karma debug page (started when you run `yarn test:dev`).
    *
    *  ## debugging
    *

tasks/setup_kibana.js

@@ -1,28 +0,0 @@
-const exec = require('child_process').execFileSync;
-const stat = require('fs').statSync;
-
-const fromRoot = require('path').resolve.bind(null, __dirname, '../');
-
-module.exports = function (grunt) {
-  grunt.registerTask('setup_kibana', function () {
-    const kbnDir = fromRoot('../kibana');
-    const kbnGitDir = fromRoot('../kibana/.git');
-
-    try {
-      if (stat(kbnGitDir).isDirectory()) {
-        exec('git', ['pull', 'origin', 'master'], { cwd: kbnDir });
-      } else {
-        throw new Error(`${kbnGitDir} is not a directory??`);
-      }
-    } catch (error) {
-      if (error.code === 'ENOENT') {
-        exec('git', ['clone', 'https://github.com/elastic/kibana.git', kbnDir]);
-      } else {
-        throw error;
-      }
-    }
-
-    exec('npm', ['prune'], { cwd: kbnDir });
-    exec('npm', ['install'], { cwd: kbnDir });
-  });
-};

tasks/vendor/README.md

@@ -0,0 +1,14 @@
+Kibana ships with a vendored version of Yarn, which is used to install
+dependencies when preparing the production build of Kibana.
+
+The currently vendored version of Yarn contains the bugfix in
+https://github.com/yarnpkg/yarn/pull/5059, which handles optional dependencies.
+
+To build a new Yarn bundle, check out that pull request and run:
+
+```
+yarn build-bundle
+```
+
+This will create the file `artifacts/yarn-1.3.2.js`, which can be renamed and
+moved to this folder.

test/scripts/jenkins_docs.sh

@@ -3,4 +3,4 @@
 set -e
 source "$(dirname $0)/../../src/dev/ci_setup/setup.sh"
 
-"$(npm bin)/grunt" jenkins:docs;
+"$(yarn bin)/grunt" jenkins:docs;

test/scripts/jenkins_selenium.sh

@@ -3,6 +3,6 @@
 set -e
 source "$(dirname $0)/../../src/dev/ci_setup/setup.sh"
 
-"$(npm bin)/grunt" build --release;
+"$(yarn bin)/grunt" build --release;
 
-xvfb-run "$(npm bin)/grunt" jenkins:selenium;
+xvfb-run "$(yarn bin)/grunt" jenkins:selenium;

test/scripts/jenkins_unit.sh

@@ -3,4 +3,4 @@
 set -e
 source "$(dirname $0)/../../src/dev/ci_setup/setup.sh"
 
-xvfb-run "$(npm bin)/grunt" jenkins:unit;
+xvfb-run "$(yarn bin)/grunt" jenkins:unit;

ui_framework/README.md

@@ -10,7 +10,7 @@
 Compile the CSS with `./node_modules/grunt/bin/grunt uiFramework:compileCss` (OS X) or
 `.\node_modules\grunt\bin\grunt uiFramework:compileCss` (Windows).
 
-You can view interactive documentation by running `npm run uiFramework:start` and then visiting
+You can view interactive documentation by running `yarn uiFramework:start` and then visiting
 [http://localhost:8020/](http://localhost:8020/). This will also start watching the SCSS files, and will recompile the CSS
 automatically for you when you make changes.
 
@@ -36,7 +36,7 @@ You can do this using Yeoman (the easy way), or you can do it manually (the hard
 
 #### Create a new component
 
-From the command line, run `npm run uiFramework:createComponent`.
+From the command line, run `yarn uiFramework:createComponent`.
 
 First, you'll be prompted for what kind of component to create:
 
@@ -127,7 +127,7 @@ content.
 
 You can use the same Yeoman generator referenced above to create documentation.
 
-From the command line, run `npm run uiFramework:documentComponent`.
+From the command line, run `yarn uiFramework:documentComponent`.
 
 First, you'll be prompted for what kind of documentation to create:
 
@@ -208,4 +208,4 @@ article and the links it has at the bottom: ["Bootstrap Bankruptcy"](http://www.
 * [Salesforce Lightning Design System](https://www.lightningdesignsystem.com/)
 * [Refills](http://refills.bourbon.io/)
 * [Formstone](https://formstone.it/)
-* [Element VueJS Framework](http://element.eleme.io/#/en-US/component/dialog)
+* [Element VueJS Framework](http://element.eleme.io/#/en-US/component/dialog)