High level walkthrough of repo folder structure (#114293)

* Create code_walkthrough.mdx

* Remove unused utilities folder and mention of it in the repo structure doc.

* Update dev_docs/contributing/code_walkthrough.mdx

Co-authored-by: Luke Elmers <lukeelmers@gmail.com>

* Update dev_docs/contributing/code_walkthrough.mdx

Co-authored-by: Luke Elmers <lukeelmers@gmail.com>

* Update dev_docs/contributing/code_walkthrough.mdx

omg my grammar is horrible.

Co-authored-by: Luke Elmers <lukeelmers@gmail.com>

* Update dev_docs/contributing/code_walkthrough.mdx

Co-authored-by: Luke Elmers <lukeelmers@gmail.com>

* Update dev_docs/contributing/code_walkthrough.mdx

Co-authored-by: Luke Elmers <lukeelmers@gmail.com>

* Update dev_docs/contributing/code_walkthrough.mdx

Co-authored-by: Luke Elmers <lukeelmers@gmail.com>

* Update dev_docs/contributing/code_walkthrough.mdx

Co-authored-by: Luke Elmers <lukeelmers@gmail.com>

* review updates

* Update dev_docs/contributing/code_walkthrough.mdx

Co-authored-by: Brandon Kobel <brandon.kobel@gmail.com>

* Update dev_docs/contributing/code_walkthrough.mdx

Co-authored-by: Brandon Kobel <brandon.kobel@gmail.com>

Co-authored-by: Luke Elmers <lukeelmers@gmail.com>
Co-authored-by: Brandon Kobel <brandon.kobel@gmail.com>
This commit is contained in:
Stacey Gammon 2021-10-07 17:42:05 -04:00 committed by GitHub
parent a5a8bb29d6
commit 0442c87591
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
3 changed files with 139 additions and 438 deletions

View file

@ -0,0 +1,139 @@
---
id: kibRepoStructure
slug: /kibana-dev-docs/contributing/repo-structure
title: Repository structure
summary: High level walk-through of our repository structure.
date: 2021-10-07
tags: ['contributor', 'dev', 'github', 'getting started', 'onboarding', 'kibana']
---
A high-level walk through of the folder structure of our [repository](https://github.com/elastic/kibana).
Tip: Look for a `README.md` in a folder to learn about its contents.
## [.buildkite](https://github.com/elastic/kibana/tree/master/.buildkite)
Managed by the operations team to set up a new buildkite ci system. Can be ignored by folks outside the Operations team.
## [.ci](https://github.com/elastic/kibana/tree/master/.ci)
Managed by the operations team to contain Jenkins settings. Can be ignored by folks outside the Operations team.
## [.github](https://github.com/elastic/kibana/tree/master/.github)
Contains GitHub configuration settings. This file contains issue templates, and the [CODEOWNERS](https://github.com/elastic/kibana/blob/master/.github/CODEOWNERS) file. It's important for teams to keep the CODEOWNERS file up-to-date so the right team is pinged for a code owner review on PRs that edit certain files. Note that the `CODEOWNERS` file only exists on the main/master branch, and is not backported to other branches in the repo.
## [api_docs](https://github.com/elastic/kibana/tree/master/api_docs)
Every file in here is auto-generated by the <DocLink id="kibDevDocsApiWelcome" text="API Docs system"/> and used to render our API documentation. If you edit a public plugin or package API and run `node scripts/build_api_docs` you will see files changed in this folder. Do not edit the contents of this folder directly!
Note that currently you may see _a lot_ of changes because that command is not run on every PR and so the content in this folder is often outdated.
## [config](https://github.com/elastic/kibana/tree/master/config)
This contains the base configuration file, `kibana.yml`. If you want to tweak any settings, create a `kibana.dev.yml` in here which will get picked up during development, and not checked into GitHub.
## [docs](https://github.com/elastic/kibana/tree/master/docs)
Every folder in here _except_ the [development one](https://github.com/elastic/kibana/tree/master/docs/development) contains manually generated asciidocs that end up hosted in our [Elastic guide](https://www.elastic.co/guide/).
The `development` folder contains markdown that is auto-generated with our legacy API docs tool. We are aiming to remove it shortly after 8.0FF.
## [dev_docs](https://github.com/elastic/kibana/tree/master/dev_docs)
This is where a lot of manually written content for our Developer Guide resides. Developers may also keep the information closer to what it's describing, but it's a good spot for high-level information, or information that describes how multiple plugins work together.
## [examples](https://github.com/elastic/kibana/tree/master/examples)
These are our tested example plugins that also get built and hosted [here](https://demo.kibana.dev/8.0/app/developerExamples). If a plugin is written for testing purposes only, it won't go in here. These example plugins should be written with the intention of helping developers understand how to use our services.
## [legacy_rfcs](https://github.com/elastic/kibana/tree/master/legacy_rfcs)
We used to write RFCs in `md` format and keep them in the repository, but we have since moved to Google Docs. We kept the folder around, since some folks still read these old docs. If you are an internal contributor you can visit <DocLink id="ktRFCProcess" text="this document"/> to read about our current RFC process.
## [licenses](https://github.com/elastic/kibana/tree/master/licenses)
Contains our two license header texts, one for the Elastic license and one for the Elastic+SSPL license. All code files inside x-pack should have the Elastic license text at the top, all code files outside x-pack should contain the other. If you have your environment set up to auto-fix on save, eslint should take care of adding it for you. If you don't have it, ci will fail. Can be ignored for the most part, this rarely changes.
## [packages](https://github.com/elastic/kibana/tree/master/packages)
The packages folder contains a mixture of build-time related code (like the [code needed to build the api docs](https://github.com/elastic/kibana/tree/master/packages/kbn-docs-utils)), as well as static code that some plugins rely on (like the [kbn-monaco package](https://github.com/elastic/kibana/tree/master/packages/kbn-monaco)). <DocLink id="kibPlatformIntro" section="1000-foot-view" text="This document"/> covers how packages differ from plugins.
## [plugins](https://github.com/elastic/kibana/tree/master/plugins)
This is an empty folder in GitHub. It's where third party developers should put their plugin folders. Internal developers can ignore this folder.
## [scripts](https://github.com/elastic/kibana/tree/master/scripts)
Contains a bunch of developer scripts. These are usually very small files with just two lines that kick off a command, the logic of which resides elsewhere (sometimes `src/dev`, sometimes inside `packages`).
Example:
```
require('../src/setup_node_env');
require('@kbn/es-archiver').runCli();
```
## [src](https://github.com/elastic/kibana/tree/master/src)
This folder and the packages folder contain the most code and where developers usually find themselves. I'll touch on a few of the subfolder, the rest can generally be ignored, or are build/ops related code.
### [src/cli*](https://github.com/elastic/kibana/tree/master/src/cli)
Maintained primarily by the Operations team, this folder contains code that initializes the Kibana runtime and a bit of work to handle authorization for interactive setup mode. Most devs should be able to ignore this code.
### [src/core](https://github.com/elastic/kibana/tree/master/src/core)
This code primarily belongs to the Core team and contains the plugin infrastructure, as well as a bunch of fundamental services like migrations, saved objects, and some UI utilities (toasts, flyouts, etc.).
### [src/dev](https://github.com/elastic/kibana/tree/master/src/dev)
Maintained by the Operations team, this code contains build and development tooling related code. This folder existed before `packages`, so contains mostly older code that hasn't been migrated to packages. Prefer creating a `package` if possible. Can be ignored for the most part if you are not on the Ops team. Prefer
### [src/plugins](https://github.com/elastic/kibana/tree/master/src/plugins)
Contains all of our Basic-licensed plugins. Most folders in this directory will contain `README.md` files explaining what they do. If there are none, you can look at the `owner.gitHub` field inside all `kibana.json`s that will tell you which team to get in touch with for questions.
Note that as plugins can be nested, each folder in this directory may contain multiple plugins.
## [test](https://github.com/elastic/kibana/tree/master/test)
Contains functional tests and related FTR (functional test runner) code for the plugins inside `src/plugins`, although there is a push to move the tests to reside inside the plugins themselves.
## [typings](https://github.com/elastic/kibana/tree/master/typings)
Maintained by Ops and Core, this contains global typings for dependencies that do not provide their own types, or don't have types available via [DefinitelyTyped](https://definitelytyped.org). This directory is intended to be minimal; types should only be added here as a last resort.
## [vars](https://github.com/elastic/kibana/tree/master/vars)
A bunch of groovy scripts maintained by the Operations team.
## [x-pack](https://github.com/elastic/kibana/tree/master/x-pack)
Contains all code and infrasturcture that powers our gold+ (non-basic) features that are provided under a more restrictive license.
### [x-pack/build_chromium](https://github.com/elastic/kibana/tree/master/x-pack/build_chromium)
Maintained by the App Services UX team, this contains Reporting-related code for building Chromium in order to take server side screenshots.
### [x-pack/dev-tools](https://github.com/elastic/kibana/tree/master/x-pack/dev-tools)
Maintained by the Operations team.
### [x-pack/examples](https://github.com/elastic/kibana/tree/master/x-pack/examples)
Contains all example plugins that rely on gold+ features.
### [x-pack/plugins](https://github.com/elastic/kibana/tree/master/x-pack/plugins)
Contains code for all the plugins that power our gold+ features.
### [x-pack/scripts](https://github.com/elastic/kibana/tree/master/x-pack/scripts)
Maintained by the Ops team, this folder contains some scripts for running x-pack utilities, like the functional test runner that runs with a license higher than Basic.
### [x-pack/test](https://github.com/elastic/kibana/tree/master/x-pack/test)
Functional tests for our gold+ features.

View file

@ -1,297 +0,0 @@
<!DOCTYPE html>
<html lang="en" xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Kibana Visual Regression Gallery</title>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<style>
body {
padding: 40px;
background-color: #f6f6f6;
font-family: 'Helvetica', Arial, sans-serif;
min-width: 2000px;
}
.title {
margin-bottom: 6px;
font-size: 28px;
}
.meta {
font-size: 14px;
margin-bottom: 30px;
}
.comparison {
padding: 20px;
margin-bottom: 40px;
background-color: #ffffff;
}
.comparison__title {
cursor: pointer;
padding-bottom: 10px;
margin-bottom: 10px;
font-size: 32px;
color: #2d2d2d;
border-bottom: 2px solid #ecf0f1;
}
.comparison__title:hover {
background-color: #e9e9e9;
}
.comparison__percent {
opacity: 0.4;
}
.comparison__subTitle {
display: -webkit-box;
display: -ms-flexbox;
display: flex;
padding-bottom: 10px;
font-weight: 700;
font-size: 14px;
color: #9c9c9c;
border-bottom: 1px solid #ecf0f1;
}
.comparison__body {
display: -webkit-box;
display: -ms-flexbox;
display: flex;
padding-top: 20px;
max-height: 500px;
overflow: auto;
}
.comparison--collapsed .comparison__title {
margin-bottom: 0;
border-bottom: 0;
}
.comparison--collapsed .comparison__subTitle {
display: none;
}
.comparison--collapsed .comparison__body {
display: none;
}
.comparison--warning {
}
.comparisonScreenshot {
}
.comparisonScreenshot + .comparisonScreenshot {
margin-left: 20px;
}
.comparisonScreenshot__image {
cursor: pointer;
width: 900px;
margin-bottom: 10px;
box-shadow: 0 0 5px rgba(0, 0, 0, 0.3);
}
.comparisonScreenshotTitle {
width: 900px;
background-color: white;
}
.comparisonScreenshotTitle + .comparisonScreenshotTitle {
margin-left: 20px;
}
.comparisonScreenshotTitle--session,
.comparisonScreenshotTitle--baseline {
cursor: pointer;
}
.comparisonScreenshotTitle--session:hover,
.comparisonScreenshotTitle--baseline:hover {
background-color: #e9e9e9;
}
</style>
</head>
<body>
<div class="title">
Kibana Visual Regression Gallery
</div>
<div class="meta">
{{branch}} - {{date}}
</div>
{{#each comparisons as |comparison|}}
<div class="comparison{{#lte comparison.change ../hiddenThreshold}} comparison--collapsed{{/lte}} {{#gte comparison.change ../warningThreshold }} comparison--warning{{/gte}}">
<div class="comparison__title">
<span class="comparison__percent">({{comparison.percentage}}%)</span> {{comparison.name}}
</div>
<div class="comparison__subTitle">
<div class="comparisonScreenshotTitle">Diff</div>
<div
class="comparisonScreenshotTitle comparisonScreenshotTitle--session"
style="display: none"
>
Session
</div>
<div
class="comparisonScreenshotTitle comparisonScreenshotTitle--baseline"
>
Baseline
</div>
</div>
<div class="comparison__body">
<!-- Diff first, so we can scan for regressions. -->
<div class="comparisonScreenshot">
<img
class="comparisonScreenshot__image"
src="data:image/png;base64,{{comparison.imageData.diff}}"
/>
</div>
<!-- Session, see what happened during the test. -->
<div
class="comparisonScreenshot comparisonScreenshot--session"
style="display: none"
>
<img
class="comparisonScreenshot__image"
src="data:image/png;base64,{{comparison.imageData.session}}"
/>
</div>
<!-- Baseline, see what it's supposed to be. -->
<div class="comparisonScreenshot comparisonScreenshot--baseline">
<img
class="comparisonScreenshot__image"
src="data:image/png;base64,{{comparison.imageData.baseline}}"
/>
</div>
</div>
</div>
{{/each}}
<script>
function start() {
function addClass(el, className) {
el.className += ' ' + className;
}
function removeClass(el, className) {
var classes = el.className.split(' ');
var classIndex = classes.indexOf(className);
if (classIndex !== -1) {
classes.splice(classIndex, 1);
}
el.className = classes.join(' ');
}
function onClickComparisonTitle(comparison) {
return function() {
var collapsedClass = 'comparison--collapsed';
if (comparison.className.indexOf(collapsedClass) === -1) {
addClass(comparison, collapsedClass);
} else {
removeClass(comparison, collapsedClass);
}
};
}
function onClickBaselineTitle(
baselineImage,
baselineTitle,
sessionImage,
sessionTitle
) {
return function() {
// Hide the baseline image.
baselineImage.setAttribute('style', 'display: none');
baselineTitle.setAttribute('style', 'display: none');
// Show the session image.
sessionImage.setAttribute('style', '');
sessionTitle.setAttribute('style', '');
}
}
function onClickSessionTitle(
baselineImage,
baselineTitle,
sessionImage,
sessionTitle
) {
return function() {
// Show the baseline image.
baselineImage.setAttribute('style', '');
baselineTitle.setAttribute('style', '');
// Hide the session image.
sessionImage.setAttribute('style', 'display: none');
sessionTitle.setAttribute('style', 'display: none');
}
}
function onClickImage(image) {
return function() {
const url = image.getAttribute('src');
window.open(url, '_blank');
}
}
var comparisons = document.querySelectorAll('.comparison');
for (var i = 0; i < comparisons.length; i++) {
var comparison = comparisons[i];
var comparisonTitle = comparison.querySelector('.comparison__title');
comparisonTitle.onclick = onClickComparisonTitle(comparison);
var baselineScreenshot = comparison.querySelector('.comparisonScreenshot--baseline');
var baselineTitle = comparison.querySelector('.comparisonScreenshotTitle--baseline');
var sessionScreenshot = comparison.querySelector('.comparisonScreenshot--session');
var sessionTitle = comparison.querySelector('.comparisonScreenshotTitle--session');
baselineTitle.onclick = onClickBaselineTitle(
baselineScreenshot,
baselineTitle,
sessionScreenshot,
sessionTitle
);
sessionTitle.onclick = onClickSessionTitle(
baselineScreenshot,
baselineTitle,
sessionScreenshot,
sessionTitle
);
const baselineImage = baselineScreenshot.querySelector('img');
baselineImage.onclick = onClickImage(baselineImage);
const sessionImage = sessionScreenshot.querySelector('img');
sessionImage.onclick = onClickImage(sessionImage);
}
}
document.onreadystatechange = function () {
if (document.readyState === 'complete') {
start();
}
};
</script>
</body>
</html>

View file

@ -1,141 +0,0 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0 and the Server Side Public License, v 1; you may not use this file except
* in compliance with, at your election, the Elastic License 2.0 or the Server
* Side Public License, v 1.
*/
import bluebird, { promisify } from 'bluebird';
import Handlebars from 'handlebars';
import fs from 'fs';
import path from 'path';
import { PNG } from 'pngjs';
import pixelmatch from 'pixelmatch';
import moment from 'moment';
import SimpleGit from 'simple-git';
const readDirAsync = promisify(fs.readdir);
const readFileAsync = promisify(fs.readFile);
const writeFileAsync = promisify(fs.writeFile);
Handlebars.registerHelper('lte', function lessThanEquals(value, threshold, options) {
if (value <= threshold) {
return options.fn(this);
}
return options.inverse(this);
});
Handlebars.registerHelper('gte', function greaterThanEquals(value, threshold, options) {
if (value >= threshold) {
return options.fn(this);
}
return options.inverse(this);
});
async function buildGallery(comparisons) {
const simpleGit = new SimpleGit();
const asyncBranch = promisify(simpleGit.branch, simpleGit);
const branch = await asyncBranch();
const template = Handlebars.compile(
await readFileAsync(
path.resolve('./utilities/templates/visual_regression_gallery.handlebars'),
'utf8'
),
{ knownHelpersOnly: true }
);
const html = template({
date: moment().format('MMMM Do YYYY, h:mm:ss a'),
branch: branch.current,
hiddenThreshold: 0,
warningThreshold: 0.03,
comparisons,
});
return writeFileAsync(
path.resolve('./test/functional/screenshots/visual_regression_gallery.html'),
html
);
}
async function compareScreenshots() {
const SCREENSHOTS_DIR = 'test/functional/screenshots';
const BASELINE_SCREENSHOTS_DIR = path.resolve(SCREENSHOTS_DIR, 'baseline');
const DIFF_SCREENSHOTS_DIR = path.resolve(SCREENSHOTS_DIR, 'diff');
const SESSION_SCREENSHOTS_DIR = path.resolve(SCREENSHOTS_DIR, 'session');
// We don't need to create the baseline dir because it's committed.
fs.mkdirSync(DIFF_SCREENSHOTS_DIR, { recursive: true });
fs.mkdirSync(SESSION_SCREENSHOTS_DIR, { recursive: true });
const files = await readDirAsync(SESSION_SCREENSHOTS_DIR);
const screenshots = files.filter((file) => file.indexOf('.png') !== -1);
// We'll use this data to build a screenshot gallery in HTML.
return await bluebird.map(screenshots, async (screenshot) => {
// We're going to load image data and cache it in this object.
const comparison = {
name: screenshot,
change: undefined,
percentage: undefined,
imageData: {
session: undefined,
baseline: undefined,
diff: undefined,
},
};
const sessionImagePath = path.resolve(SESSION_SCREENSHOTS_DIR, screenshot);
const baselineImagePath = path.resolve(BASELINE_SCREENSHOTS_DIR, screenshot);
const diffImagePath = path.resolve(DIFF_SCREENSHOTS_DIR, screenshot);
const sessionImage = PNG.sync.read(await readFileAsync(sessionImagePath));
const baselineImage = PNG.sync.read(await readFileAsync(baselineImagePath));
const { width, height } = sessionImage;
const diff = new PNG({ width, height });
const numDiffPixels = pixelmatch(
sessionImage.data,
baselineImage.data,
diff.data,
width,
height,
{ threshold: 0 }
);
await writeFileAsync(diffImagePath, PNG.sync.write(diff));
const change = numDiffPixels / (width * height);
const changePercentage = (change * 100).toFixed(2);
console.log(`(${changePercentage}%) ${screenshot}`);
comparison.percentage = changePercentage;
comparison.change = change;
// Once the images have been diffed, we can load and store the image data.
comparison.imageData.session = await readFileAsync(sessionImagePath, 'base64');
comparison.imageData.baseline = await readFileAsync(baselineImagePath, 'base64');
comparison.imageData.diff = await readFileAsync(diffImagePath, 'base64');
return comparison;
});
}
export function run(done) {
compareScreenshots().then(
(screenshotComparisons) => {
// Once all of the data has been loaded, we can build the gallery.
buildGallery(screenshotComparisons).then(() => {
done();
});
},
(error) => {
console.error(error);
done(false);
}
);
}