The EBT pacakge has been moved to a separate NPM package
([@elastic/ebt](https://www.npmjs.com/package/@elastic/ebt))
The npm package is on version `0.0.x` until we finish the reviews then
i'll publish the `1.0.0` version before merging this PR.
The PR is mostly code deletes after moving the code to the public ebt
github repo https://github.com/elastic/ebt
The significant changes are:
1. removed the `packages/analytics/ebt` package from kibana
2. remove @kbn/ebt references in favor of the npm package.
3. Added a util package to provide the package with the telemetry
endpoint and headers
This was previously backed into the package but now i've rewired it be
provided from Kibana, this way we have more control over the URL and
headers we use to send EBT telemetry for our elastic endpoint, which
will probably be different between users of this package and this way
we'll also avoid republishing the package if we ever want to change
these details.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Alejandro Fernández Haro <afharo@gmail.com>
## Summary
At the moment, our package generator creates all packages with the type
`shared-common`. This means that we cannot enforce boundaries between
server-side-only code and the browser, and vice-versa.
- [x] I started fixing `packages/core/*`
- [x] It took me to fixing `src/core/` type to be identified by the
`plugin` pattern (`public` and `server` directories) vs. a package
(either common, or single-scoped)
- [x] Unsurprisingly, this extended to packages importing core packages
hitting the boundaries eslint rules. And other packages importing the
latter.
- [x] Also a bunch of `common` logic that shouldn't be so _common_ 🙃
### For maintainers
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
* Adds the ability to exclude a response schema when declaring route
schemas
* Adds the ability to provide a description of a the response
See code comments for more info.
## Example
You can declare a response with no validation to imply an empty object
in OAS
```
router.versioned.post({ version: '2023-10-31', access: 'public', path: '...' })
.addVersion({
validation: {
responses: {
201: { description: 'Resource created!' }
}
}
}, () => {})
```
Will result in
```jsonc
{
//...
201: { description: 'Resource created!' }
//...
}
```
## Risks
No notable risks
## Summary
Per [the OAS docs](https://swagger.io/specification/), they have an info
object with a `summary` and `description` field. This PR refactors the
existing router `description` field to to OAS `summary` (that is how it
has been used) and introduces a "new" `description` field that will be
used for the longer form descriptions.
## Resources
* https://swagger.io/specification/
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: lcawl <lcawley@elastic.co>
## Summary
Close https://github.com/elastic/kibana/issues/182910
Add the ability to declare recursive schemas. Updates
`@kbn/config-schema` to support recursive types. This design follows the
underlying pattern provided by Joi:
https://joi.dev/api/?v=17.13.0#linkref:
```ts
const id = 'recursive';
const recursiveSchema: Type<RecursiveType> = schema.object(
{
name: schema.string(),
self: schema.lazy<RecursiveType>(id),
},
{ meta: { id } }
);
```
Since using the `.link` API we are also using `.id` which enables us to
leverage this mechanism OOTB with `joi-to-json` for OAS generation (thus
we could delete a lot of code there).
I chose to avoid using `id` originally because I thought it would be
simpler if we control more of the conversion in config-schema's case but
for recursive schemas and references I think this is a favourable trade
off. Open to other ideas!
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
We were spending a lot of time in `calculateDepthRecursive`
<img width="1367" alt="Screenshot 2024-03-21 at 13 46 00"
src="11047c7b-9373-4976-ba40-8e6ad8d15373">
And it could fully be avoided given the plugin service already sorts the
plugins.
This PR changes the way the plugin system exposes its
`PluginDependencies` so that the keys are topologically ordered (same
order we use to setup / start the plugins) and then use that ordering
directly from the status service.
## Summary
Fix https://github.com/elastic/kibana/issues/176475
Add the `version.build_flavor` field to the response of the status API
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Addresses https://github.com/elastic/kibana-team/issues/697
The endpoint is currently waiting for all statuses to be reported.
[Recent changes](https://github.com/elastic/kibana/pull/172268) are
causing the `pluginsStatus$` Observable to take longer to emit first
value.
This PR injects a "status not reported" to prevent the different status
Observables from blocking calls to the `/api/status` enpdoint.
## Summary
The _plugins status service_ listens to some of the plugins' statuses
through Observables.
If these plugins don't emit an initial status after some time, it
injects a first "timed out" value, with an `unavailable` state.
We're currently starting this 30s timeout at service creation time.
This is not accurate, as other services starting before plugins (e.g.
saved objects + migrations) are time consuming and can leave short to no
time for plugins to `start()` before hitting the 30s timeout.
This PR aims at fixing this, by starting to count when the plugins
`start()` methods are called.
This way, we're actually giving plugins 30s to emit a status.
## Summary
New attempt at fixing https://github.com/elastic/kibana/issues/116718
Inspired on https://github.com/elastic/kibana/pull/126320
Here's what the newly logged `[status]` information looks like on a
fresh startup:
<img width="1834" alt="image"
src="d78d7f88-139f-4daf-9dc0-c4e6724ea412">
The first 2 entries are logs from Core services 🆕 .
The next 5 entries are emitted due to `taskManager` plugin emitting a
degraded status right at startup.
I have created an issue to tackle that one:
https://github.com/elastic/kibana/issues/168237
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Allow Kibana to restrict the usage of JWT for a predefined set of routes
only in Serverless environment by default. This capability is not
available in non-Serverless environment.
Any route that needs to be accessed in Serverless environemnt using JWT
as a means of authentication should include `security:acceptJWT` tag.
## How to test
If you'd like to generate your own JWT to test the PR, please follow the
steps outlined in
https://github.com/elastic/kibana/pull/159117#issue-1743796706 or just
run functional test server and use static JWT from the Serverless test.
This PR also generated a Serverless Docker image that you can use in
your Dev/QA MKI cluster.
- [x] Implementation functionality and add unit tests
- [x] Update metrics/status routes to include new `security:acceptJWT`
tag
- [x] Update serverless test suite to include a test for
`security:acceptJWT`
__Fixes: https://github.com/elastic/kibana/issues/162632__
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
When turning on `server.restrictInternalApis` a number of issues
surfaced due to defaulting to internal resulting in `400`s for:
* HTTP resources
* Static assets via `registerStaticDir`
* Use of `res.render(Html|Js|Css)` outside of HTTP resources
This PR:
* defaults our HTTP resources service to register routes by default
`public`, same for static dirs.
* Did an audit of all renderX usages, if outside of HTTP resources I
added an explicit `access: public`
* ...what else?
### Set `access: 'public'` for known set of "system" routes
Method | Path | Comment
-- | -- | --
GET | /api/status
GET | /api/stats
GET | /translations/{locale}.json
GET | /api/fleet/agent_policies
GET | /api/task_manager/_background_task_utilization
GET | /internal/task_manager/_background_task_utilization
GET | /internal/detection_engine/health/_cluster
POST | /internal/detection_engine/health/_cluster
GET | /internal/detection_engine/health/_space
POST | /internal/detection_engine/health/_space
POST | /internal/detection_engine/health/_rule
POST | /internal/detection_engine/health/_setup
GET | /bootstrap.js
GET | /bootstrap-anonymous.js
GET | \*\*/bundles/\* | Core's routes for serving JS & CSS bundles
## How to test
Run this PR with `kibana.dev.yml` containing
`server.restrictInternalApis: true` and navigate around Kibana UI
checking that there are no `400`s in the network resources tab due to
access restrictions.
## Notes
* Either left a comment about why `access` was set public or a simple
unit test to check that we are setting access for a given route
## To do
- [x] Manually test Kibana
- [x] Manually test with `interactiveSetup` plugin
- [ ] Add integration and e2e test (will do in a follow up PR)
Related: https://github.com/elastic/kibana/pull/162149
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fix https://github.com/elastic/kibana/issues/158910
Changes the behavior of the `/api/status` endpoint to always returns a
consistent http status code, and in particular:
- during the preboot stage
- when accessed by unauthenticated users and `status.allowAnonymous` is
`false`.
That way, `/api/status` can properly be used for readiness checks.
Please refer to https://github.com/elastic/kibana/issues/158910 for more
details.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Adds build date to `GET kbn:api/status` similar to ES. Example output
running locally:
```json
{
"name": "ES-DMVD5M3",
"uuid": "545ba70c-063e-449b-af21-6c8e7b30f77e",
"version": {
"number": "8.9.0",
"build_hash": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
"build_number": 9007199254740991,
"build_snapshot": false,
"build_date": "2023-05-15T23:12:09.000Z"
},
...rest
}
```
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Release Note
The status endpoint now returns the build date alongside other build
information.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Dearest Reviewers 👋
I've been working on this branch with @mistic and @tylersmalley and
we're really confident in these changes. Additionally, this changes code
in nearly every package in the repo so we don't plan to wait for reviews
to get in before merging this. If you'd like to have a concern
addressed, please feel free to leave a review, but assuming that nobody
raises a blocker in the next 24 hours we plan to merge this EOD pacific
tomorrow, 12/22.
We'll be paying close attention to any issues this causes after merging
and work on getting those fixed ASAP. 🚀
---
The operations team is not confident that we'll have the time to achieve
what we originally set out to accomplish by moving to Bazel with the
time and resources we have available. We have also bought ourselves some
headroom with improvements to babel-register, optimizer caching, and
typescript project structure.
In order to make sure we deliver packages as quickly as possible (many
teams really want them), with a usable and familiar developer
experience, this PR removes Bazel for building packages in favor of
using the same JIT transpilation we use for plugins.
Additionally, packages now use `kbn_references` (again, just copying the
dx from plugins to packages).
Because of the complex relationships between packages/plugins and in
order to prepare ourselves for automatic dependency detection tools we
plan to use in the future, this PR also introduces a "TS Project Linter"
which will validate that every tsconfig.json file meets a few
requirements:
1. the chain of base config files extended by each config includes
`tsconfig.base.json` and not `tsconfig.json`
1. the `include` config is used, and not `files`
2. the `exclude` config includes `target/**/*`
3. the `outDir` compiler option is specified as `target/types`
1. none of these compiler options are specified: `declaration`,
`declarationMap`, `emitDeclarationOnly`, `skipLibCheck`, `target`,
`paths`
4. all references to other packages/plugins use their pkg id, ie:
```js
// valid
{
"kbn_references": ["@kbn/core"]
}
// not valid
{
"kbn_references": [{ "path": "../../../src/core/tsconfig.json" }]
}
```
5. only packages/plugins which are imported somewhere in the ts code are
listed in `kbn_references`
This linter is not only validating all of the tsconfig.json files, but
it also will fix these config files to deal with just about any
violation that can be produced. Just run `node scripts/ts_project_linter
--fix` locally to apply these fixes, or let CI take care of
automatically fixing things and pushing the changes to your PR.
> **Example:** [`64e93e5`
(#146212)](64e93e5806)
When I merged main into my PR it included a change which removed the
`@kbn/core-injected-metadata-browser` package. After resolving the
conflicts I missed a few tsconfig files which included references to the
now removed package. The TS Project Linter identified that these
references were removed from the code and pushed a change to the PR to
remove them from the tsconfig.json files.
## No bazel? Does that mean no packages??
Nope! We're still doing packages but we're pretty sure now that we won't
be using Bazel to accomplish the 'distributed caching' and 'change-based
tasks' portions of the packages project.
This PR actually makes packages much easier to work with and will be
followed up with the bundling benefits described by the original
packages RFC. Then we'll work on documentation and advocacy for using
packages for any and all new code.
We're pretty confident that implementing distributed caching and
change-based tasks will be necessary in the future, but because of
recent improvements in the repo we think we can live without them for
**at least** a year.
## Wait, there are still BUILD.bazel files in the repo
Yes, there are still three webpack bundles which are built by Bazel: the
`@kbn/ui-shared-deps-npm` DLL, `@kbn/ui-shared-deps-src` externals, and
the `@kbn/monaco` workers. These three webpack bundles are still created
during bootstrap and remotely cached using bazel. The next phase of this
project is to figure out how to get the package bundling features
described in the RFC with the current optimizer, and we expect these
bundles to go away then. Until then any package that is used in those
three bundles still needs to have a BUILD.bazel file so that they can be
referenced by the remaining webpack builds.
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* create empty packages
* start moving things to packages
* move status types to common package
* create internal common package
* start adapting usages
* more import fixes
* fix more usages
* some test file fixes
* fix some more usages
* fix test_utils import
* fix mixed import
* fix test imports
* updating READMEs
* explicit export of service setup deps
* add jsonc files to new packages
* [CI] Auto-commit changed files from 'node scripts/generate codeowners'
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
