This PR does:
- checks Kibana.spec file
- Checks server feature.ts
- Adds correct route access to APIs
- Removes unnecessary logs
- Removes collector and symbolized `secret_token` from config schema as
it won't be used
- Add README file
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Partially address https://github.com/elastic/kibana/issues/158835, add
cloud chat (drift) to more places: all management pages and
home/getting_started page
I hit an issue that both management and home couldn't depend directly on
`cloudChat` plugin. Here is the issue with more details
https://github.com/elastic/kibana/issues/159008. I worked around with
creating an intermediate `cloudChatProvider` plugin.
How do I run drift locally?
Add this to kibana.yml
```
xpack.cloud.id: "some-id"
xpack.cloud.trial_end_date: "2023-06-21T00:00:00.000Z"
xpack.cloud_integrations.chat.enabled: true
xpack.cloud_integrations.chat.chatURL: "https://elasticcloud-production-chat-us-east-1.s3.amazonaws.com/drift-iframe.html"
xpack.cloud_integrations.chat.chatIdentitySecret: "some-secret" (get it from drift console)
```
You need to have access to our drift account. But I tested with a custom
account. To change account id I had to point
`xpack.cloud_integrations.chat.chatURL` to a script with custom drift
id.
[Documentation](https://docs.google.com/document/d/1Ms8d8d_fbTTRHlBroEAKGNMNk3jFFgOAkVDRhqLxAPQ/edit?pli=1#)
issue: https://github.com/elastic/kibana/issues/158810
## Summary
This PR is a cleanup to make [this
POC](https://github.com/elastic/kibana/pull/155420) production ready
- Serverless PLI features splitting in Security Solution, to allow/deny
access to configured functionalities, using the current Kibana RBAC
service.
- Create the Upselling service to display Serveless-specific prompts in
the application when features are not available
- Create a `SecurityRoutePageWrapper` component that wraps Pages and
displays the upsell when necessary.
- We will refactor the code base to use `SecurityRoutePageWrapper`
everywhere on another PR.
- Create an Upsell page and section for entity analytics
bd8db822-2f4b-4545-9da7-bedc07d93f90
### test:
Serverless: `yarn serverless-security`.
* To change the product line you have to update
`xpack.serverless.security.productLineIds` on
`config/serverless.security.yml`.
ESS: `yarn start`
### Glossary
* PLI - Product Line Item (`Alert Triage`, `Osquery`, `Cases` , ... )
* Product Line - The product that the user is subscribed to (Security
Essentials, Security Complete, ...)
* essSecurity - New plugin with code that only runs for ESS offer
(non-serverless).
* App Feature - A security solution feature or group of features that
can be disabled for a product line. It can be mapped to PLIs (`Alert
Triage`, `Osquery`, `Cases` , ... ).
* Capability - A string that when present represents that the user can
access a given feature. A capability could be of the type UI or API
(`read_cases`, `crud_cases`, ...).
### Current architecture
### New architecture
### How does it work?
Every serverless product line (endpointEssentials, cloud essentials) can
define which features are enabled:
69d0fc15f4/x-pack/plugins/serverless_security/common/pli/pli_config.ts (L12-L19)
For ESS (non-serverless) offer we enable all features by default.
69d0fc15f4/x-pack/plugins/ess_security/server/constants.ts (L10-L13)
A feature can define privileges:
69d0fc15f4/x-pack/plugins/security_solution/server/lib/app_features/security_kibana_features.ts (L177-L185)
When the feature is enabled the privileges get merged into the base
config and injected into kibana features.
69d0fc15f4/x-pack/plugins/security_solution/server/lib/app_features/app_features.ts (L61-L70)
### TODO
- [x] lazy load these components
- [x] Add unit test to:
- ~SecurityRoutePageWrapper
x-pack/plugins/security_solution/public/common/components/security_route_page_wrapper/index.tsx~
-
~x-pack/plugins/security_solution/public/common/hooks/use_upselling.ts~
-
~x-pack/plugins/security_solution/public/common/lib/capabilities/has_capabilities.ts~
-
~x-pack/plugins/security_solution/public/common/lib/upsellings/upselling_service.ts~
- ~x-pack/plugins/serverless_security/common/pli/pli_features.ts~
-
~x-pack/plugins/serverless_security/public/components/upselling/register_upsellings.tsx~
-
~x-pack/plugins/security_solution/server/lib/app_features/app_features.ts~
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/154330
This PR:
- Moves the editor from unified-search to a standalone package
- The editor has now a core ui settings dependency but is going to have
an expressions dependency too when merged with the ESQL branch
- Adds a new plugin (text-based-languages) which is used to pass the
dependencies on the package. The user can either use this plugin without
giving any dependencies or use the package with passing the dependecies
on the KibanaContextProvider.
- Adds storybook for the editor (I used the mdx stories as we did on the
random sampling package)
<img width="1668" alt="image"
src="763a3112-1ae5-49bb-81f3-acd02892e402">
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Moves a series of Lens components to an independent plugin for reuse in
the annotations library.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] remove mentions of Lens
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Closes#154733
Creates a new plugin for logs onboarding with wizard to organize steps
into discrete views.
#### TODO:
- [x] rename plugin to observability_onboarding
- [x] configure: UI and server plugin
- [x] enable/disable new plugin
- [x] remove the link to it from Observability nav
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Yngrid Coello <yngrid.coello@elastic.co>
Co-authored-by: Yngrid Coello <yngrdyn@gmail.com>
## Summary
This plugin will contain the asset inventory and topology API in Kibana,
giving Kibana projects access to inventory and topology data via an HTTP
and/or JS API on the server and client.
[Currently proposed API
docs](https://github.com/elastic/o11y-topology-playground/tree/main/docs/api)
will be moved to this repo as well, contained inside this plugin folder,
as a part of this PR.
## Enabling the plugin
This plugin is entirely in "technical preview" and because of this, must
be specifically enabled via config for it to do anything besides being
run by the core plugin framework. To enable the server API layer, as
well as the index template management, put the following line in your
kibana.yml file:
```yml
xpack.assetManager.alphaEnabled: true
```
## Running the API integration tests
Run the functional test server with the asset manager config in place:
```shell
$ node scripts/functional_tests_server --config x-pack/test/api_integration/apis/asset_manager/config.ts
```
Then run the functional test runner with the same config, to target just
these tests:
```shell
$ node scripts/functional_test_runner --config=x-pack/test/api_integration/apis/asset_manager/config.
ts
```
_Note:_ The config file added in this folder enables the tech preview
plugin ([see file
here](https://github.com/elastic/kibana/pull/152456/files#diff-bc00de6c34c9bc131cfbdf3570c487fe9ee947e9a88a84c59d6b139b79d7708eR20)).
### Running the integration tests for verifying that the plugin is
"disabled" by default
There is a small set of tests that confirm that the endpoints return 404
and there is no index template installed if the config value is not set
in the kibana.yml file. To run this suite, use the following config:
```shell
$ node scripts/functional_tests_server --config x-pack/test/api_integration/apis/asset_manager/config_when_disabled.ts
$ node scripts/functional_test_runner --config=x-pack/test/api_integration/apis/asset_manager/config_when_disabled.
ts
```
## Testing this PR with sample data
There are some sample data mechanisms in place inside this PR to allow
us to build out the endpoints.
### View sample docs
```http
GET /api/asset-manager/assets/sample
```
This will return a list of the assets that are included if you elect to
write assets. This is a good endpoint to use to find EAN (Elastic Asset
Name) values that you may want to exclude from writing for a given time
period, to simulate assets appearing/disappearing over time.
### Write sample docs
```http
POST /api/asset-manager/assets/sample
{
"baseDateTime": "2023-02-28T12:00:00.000Z",
"excludeEans": ["k8s.cluster:cluster-002"]
}
```
This posts all of the sample asset documents to Elasticsearch using the
`baseDateTime` value as the timestamp. Any valid string or number that
is accepted by `new Date()` should work for `baseDateTime`.
The `excludeEans` value is an array of EAN ("Elastic Asset Name") values
that you don't want to write on this particular run. This way you can
have assets appear (exclude them in the past, don't exclude them during
a later run) or disappear (vice versa) and see how that shows up in
other endpoints.
**Note:** *Remember that when you curl a Kibana server API with a POST
request, you must include a `kbn-xsrf` header with any string value you
want.*
### Get asset docs from ES
```http
GET /api/asset-manager/assets?type=k8s.cluster&from=now-10m
```
This is the primary "real" endpoint available right now. It should
retrieve a list of assets based on the type/from/to/ean filter values
you specify. Once you load the sample data, this endpoint should return
results.
## Debug logging
There are some extra debug logs for ES queries that are running in the
code in this PR. To print those logs to the Kibana server console, run
Kibana using `DEBUG_LOGGER=true`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
# [Security Solution] Data Quality dashboard
## Check ECS compatibility with just one click
With just one click, the _Data Quality dashboard_ checks all the indices used by the Security Solution, (or anything else), for compatibility with the [Elastic Common Schema (ECS)](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html)
## Create cases from results
Create a single case containing all the results, or create cases for specific indices
## Interactive tabs put results in context
Expand any index to reveal interactive tabs
- Summary
- Incompatible fields
- Custom fields
- ECS complaint fields
- All fields
## Share comprehensive markdown reports
Share markdown reports containing the same content as the dashboard
### On page load
When the Data Quality dashboard page loads, the alerts index, and any indices matching the selected `Data view` are displayed
Only `hot`, `warm`, or `unmanaged` indices are displayed by default
Indices are not checked automatically when the dashboard loads
Click either :
- `Check all` to check all the indices on the page
- The expand button to automatically check (just) one index, and instantly view results
### Check all
When the `Check all` button is clicked
- The `Check all` button changes to a `Cancel` button
- The `Last checked: n <time unit> ago` text is replaced with a progress bar indicating how many Indices are left to check
- The `Checking <index name>` text will update as each index is checked. Text will wrap if necessary
- The results tables begin updating with results
- Pattern stats update to summarize each table
- Rolled up results for the entire page update after every index is checked
<https://user-images.githubusercontent.com/4459398/216007795-2ebbc0c6-8c7a-49c7-a22c-b97d2a58dddd.mov>
When Check all, is running, the Data Quality dashboard adds a three second delay after every check completes, before beginning the next check.
Check all will keep checking indexes until the user cancels, or all indexes have (attempted to be) checked.
While Check all is running, users may simultaneously click on any index to check it on demand. The results are instantly rolled up when this happens.
When all checks complete, the page looks like this:
### Take action
Click the `Take action` popover to share the entire page of results via one of the following actions:
- Add to new case
- Copy to clipboard
### Expanding results
The `Incompatible fields` tab is always displayed by default when a result is expanded
The `Incompatible fields` tab shows a success message when a successful result is expanded
The `Incompatible fields` tab shows, side by side, expected ECS mapping types vs the actual mapping types when they are different
The `Incompatible fields` tab also compares field values expected by ECS vs the actual values in an index, when they are different
The `Incompatible fields` tab displays a callout that explains the consequences of having incompatible fields. The content is based on the following illustration, created by @MikePaquette
<img width="1264" alt="ecs_meter" src="https://user-images.githubusercontent.com/4459398/216016124-6fe89ab4-c364-40ec-8a6f-99349e6d583c.png">
The calllout has a call to action to create a case or copy a markdown report for just the expanded result
- Add to new case
- Copy to clipboard
### Tabs
The Summary tab displays a call to action when incompatible fields are found
Click on any part of the Summary tab chart or legend to navigate to the corresponding tab
Clicking on the `Copy to clipboard` call to action in the Custom fields tab copies a markdown version of the table to the clipboard
The search feature of the ECS complaint fields tab may, for example, be used to verify a specific ECS complaint mapping exists
The All fields tab displays the union of all other tabs
### Data view selection
The `Data view` dropdown defaults to the `Security Default Data View`
The alerts index is always checked and included in the results, even when another Data View is selected
### ILM phase options
Only `hot`, `warm`, or `unmanaged` indices may be selected for checking.
The `cold` and `frozen` options are disabled.
When all options in the `ILM phase` box are cleared, an informative empty prompt is displayed
### Errors
Errors may occur for some (or all) indices. The `View errors` button appears when the first error occurs
Users may click the `View errors` button to view them, even while a check is in progress
The Copy to clipboard button in the errors popover copies a markdown version of the errors table to the clipboard
When errors occur, the same content shown in the Errors popover is automatically included in the markdown report created by the `Take action` menu
### Markdown reports
The content of markdown reports (created by the Take action menu) includes most of the content from the Data Quality dashboard that created it
In the screenshot below, the Data Quality dashboard is on the left, and a markdown report (pasted into Github) is on the right
Stats rollups and tables are included in markdown reports
Markdown reports use the same "expected vs actual" format to display the details of incompatible field mappings
### Navigation
The Data Quality dashboard is grouped with the existing Security Solution dashboards
It may also be launched via the side navigation
## Privileges
The privileges in the table below are required to check any pattern of indices, or any specific index:
| Privilege | Required to | Required for API |
|-------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|------------------|
| `monitor` or `manage` (`manage` builds on `monitor`) | List indices that match a pattern, and get document counts for an index example: `GET logs-*/_stats` | `_stats` |
| `view_index_metadata` or `manage_ilm` | List index ILM configs (e.g. hot) that match a pattern example: `GET logs-*/_ilm/explain` | `_ilm/explain` |
| `view_index_metadata` or `manage` | Get index mappings for a specific index example: `GET .ds-logs-endpoint.events.process-default-2023.01.17-000001/_mapping` | `_mapping` |
| `read` or `read_cross_cluster` | Run aggregations to test for unallowed values example: `GET .ds-logs-endpoint.events.process-default-2023.01.17-000001/_search` | `_search` |
Users may have some of the privileges required to check an index, but not all of them.
The built-in `viewer` role does not have the `monitor` (or `manage`) role. The following screenshot illustrates what a user will see if they login as a user with the `viewer` role:
# An actual markdown report (all content below)
The rest of the content below is pasted from an actual report, created via the `Take action` menu:
# Data quality
| Incompatible fields | Indices checked | Indices | Docs |
|---------------------|-----------------|---------|------|
| 17 | 15 | 17 | 1,404,514 |
## .alerts-security.alerts-default
`hot(1)`
| Incompatible fields | Indices checked | Indices | Docs |
|---------------------|-----------------|---------|------|
| 1 | 1 | 1 | 1,837 |
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | .internal.alerts-security.alerts-default-000001 | 1,837 (100.0%) | 1 | `hot` |
### .internal.alerts-security.alerts-default-000001
The `.internal.alerts-security.alerts-default-000001` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html).
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | .internal.alerts-security.alerts-default-000001 | 1,837 (100.0%) | 1 | `hot` |
### **Incompatible fields** `1` **Custom fields** `188` **ECS compliant fields** `1219` **All fields** `1408`
#### 1 incompatible field
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
#### Incompatible field values - .internal.alerts-security.alerts-default-000001
| Field | ECS values (expected) | Document values (actual) |
|-------|-----------------------|--------------------------|
| event.category | `authentication`, `configuration`, `database`, `driver`, `email`, `file`, `host`, `iam`, `intrusion_detection`, `malware`, `network`, `package`, `process`, `registry`, `session`, `threat`, `vulnerability`, `web` | `behavior` (62) |
## auditbeat-*
`hot(11)` `unmanaged(1)`
| Incompatible fields | Indices checked | Indices | Docs |
|---------------------|-----------------|---------|------|
| 13 | 10 | 12 | 29,182 |
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.6.0-2023.01.17-000001 | 14,409 (49.4%) | 0 | `hot` |
| -- | .ds-auditbeat-8.5.3-2023.01.24-000001 | 2,857 (9.8%) | -- | `hot` |
| ✅ | .ds-auditbeat-8.2.3-2023.01.24-000001 | 2,246 (7.7%) | 0 | `hot` |
| ✅ | .ds-auditbeat-8.4.1-2023.01.24-000001 | 2,179 (7.5%) | 0 | `hot` |
| -- | .ds-auditbeat-8.3.3-2023.01.24-000001 | 1,921 (6.6%) | -- | `hot` |
| ✅ | auditbeat-7.16.0-2023.01.17-000001 | 1,880 (6.4%) | 0 | `hot` |
| ✅ | .ds-auditbeat-8.1.1-2023.01.24-000001 | 1,676 (5.7%) | 0 | `hot` |
| ✅ | .ds-auditbeat-8.2.2-2023.01.24-000001 | 1,578 (5.4%) | 0 | `hot` |
| ✅ | .ds-auditbeat-8.0.0-2023.01.24-000001 | 251 (0.9%) | 0 | `hot` |
| ❌ | auditbeat-7.10.2-2023.01.24-000001 | 111 (0.4%) | 12 | `hot` |
| ✅ | .ds-auditbeat-8.5.0-2023.01.24-000001 | 74 (0.3%) | 0 | `hot` |
| ❌ | auditbeat-custom-empty-index-1 | 0 (0.0%) | 1 | `unmanaged` |
### .ds-auditbeat-8.6.0-2023.01.17-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.6.0-2023.01.17-000001 | 14,409 (49.4%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `549` **ECS compliant fields** `1210` **All fields** `1759`
### .ds-auditbeat-8.2.3-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.2.3-2023.01.24-000001 | 2,246 (7.7%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `510` **ECS compliant fields** `1210` **All fields** `1720`
### .ds-auditbeat-8.4.1-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.4.1-2023.01.24-000001 | 2,179 (7.5%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `509` **ECS compliant fields** `1210` **All fields** `1719`
### auditbeat-7.16.0-2023.01.17-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | auditbeat-7.16.0-2023.01.17-000001 | 1,880 (6.4%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `523` **ECS compliant fields** `1111` **All fields** `1634`
### .ds-auditbeat-8.1.1-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.1.1-2023.01.24-000001 | 1,676 (5.7%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `510` **ECS compliant fields** `1204` **All fields** `1714`
### .ds-auditbeat-8.2.2-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.2.2-2023.01.24-000001 | 1,578 (5.4%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `510` **ECS compliant fields** `1210` **All fields** `1720`
### .ds-auditbeat-8.0.0-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.0.0-2023.01.24-000001 | 251 (0.9%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `510` **ECS compliant fields** `1204` **All fields** `1714`
### auditbeat-7.10.2-2023.01.24-000001
The `auditbeat-7.10.2-2023.01.24-000001` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html).
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | auditbeat-7.10.2-2023.01.24-000001 | 111 (0.4%) | 12 | `hot` |
### **Incompatible fields** `12` **Custom fields** `467` **ECS compliant fields** `602` **All fields** `1081`
#### 12 incompatible fields
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
#### Incompatible field mappings - auditbeat-7.10.2-2023.01.24-000001
| Field | ECS mapping type (expected) | Index mapping type (actual) |
|-------|-----------------------------|-----------------------------|
| error.message | `match_only_text` | `text` |
| error.stack_trace | `wildcard` | `keyword` |
| http.request.body.content | `wildcard` | `keyword` |
| http.response.body.content | `wildcard` | `keyword` |
| message | `match_only_text` | `text` |
| process.command_line | `wildcard` | `keyword` |
| process.parent.command_line | `wildcard` | `keyword` |
| registry.data.strings | `wildcard` | `keyword` |
| url.full | `wildcard` | `keyword` |
| url.original | `wildcard` | `keyword` |
| url.path | `wildcard` | `keyword` |
#### Incompatible field values - auditbeat-7.10.2-2023.01.24-000001
| Field | ECS values (expected) | Document values (actual) |
|-------|-----------------------|--------------------------|
| event.kind | `alert`, `enrichment`, `event`, `metric`, `state`, `pipeline_error`, `signal` | `error` (1) |
### .ds-auditbeat-8.5.0-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.5.0-2023.01.24-000001 | 74 (0.3%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `509` **ECS compliant fields** `1210` **All fields** `1719`
### auditbeat-custom-empty-index-1
The `auditbeat-custom-empty-index-1` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html).
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | auditbeat-custom-empty-index-1 | 0 (0.0%) | 1 | `unmanaged` |
### **Incompatible fields** `1` **Custom fields** `0` **ECS compliant fields** `0` **All fields** `0`
#### 1 incompatible field
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
#### Incompatible field mappings - auditbeat-custom-empty-index-1
| Field | ECS mapping type (expected) | Index mapping type (actual) |
|-------|-----------------------------|-----------------------------|
| @timestamp | `date` | `-` |
## logs-*
`hot(2)`
| Incompatible fields | Indices checked | Indices | Docs |
|---------------------|-----------------|---------|------|
| 3 | 2 | 2 | 602 |
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | .ds-logs-endpoint.alerts-default-2023.01.17-000001 | 342 (56.8%) | 2 | `hot` |
| ❌ | .ds-logs-endpoint.events.process-default-2023.01.17-000001 | 260 (43.2%) | 1 | `hot` |
### .ds-logs-endpoint.alerts-default-2023.01.17-000001
The `.ds-logs-endpoint.alerts-default-2023.01.17-000001` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html).
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | .ds-logs-endpoint.alerts-default-2023.01.17-000001 | 342 (56.8%) | 2 | `hot` |
### **Incompatible fields** `2` **Custom fields** `857` **ECS compliant fields** `675` **All fields** `1534`
#### 2 incompatible fields
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
#### Incompatible field mappings - .ds-logs-endpoint.alerts-default-2023.01.17-000001
| Field | ECS mapping type (expected) | Index mapping type (actual) |
|-------|-----------------------------|-----------------------------|
| process.env_vars | `keyword` | `object` |
#### Incompatible field values - .ds-logs-endpoint.alerts-default-2023.01.17-000001
| Field | ECS values (expected) | Document values (actual) |
|-------|-----------------------|--------------------------|
| event.category | `authentication`, `configuration`, `database`, `driver`, `email`, `file`, `host`, `iam`, `intrusion_detection`, `malware`, `network`, `package`, `process`, `registry`, `session`, `threat`, `vulnerability`, `web` | `behavior` (45) |
### .ds-logs-endpoint.events.process-default-2023.01.17-000001
The `.ds-logs-endpoint.events.process-default-2023.01.17-000001` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html).
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | .ds-logs-endpoint.events.process-default-2023.01.17-000001 | 260 (43.2%) | 1 | `hot` |
### **Incompatible fields** `1` **Custom fields** `130` **ECS compliant fields** `304` **All fields** `435`
#### 1 incompatible field
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
#### Incompatible field mappings - .ds-logs-endpoint.events.process-default-2023.01.17-000001
| Field | ECS mapping type (expected) | Index mapping type (actual) |
|-------|-----------------------------|-----------------------------|
| process.env_vars | `keyword` | `object` |
## packetbeat-*
`hot(2)`
| Incompatible fields | Indices checked | Indices | Docs |
|---------------------|-----------------|---------|------|
| 0 | 2 | 2 | 1,372,893 |
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-packetbeat-8.6.0-2023.01.17-000001 | 704,062 (51.3%) | 0 | `hot` |
| ✅ | .ds-packetbeat-8.4.1-2023.01.24-000001 | 668,831 (48.7%) | 0 | `hot` |
### .ds-packetbeat-8.6.0-2023.01.17-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-packetbeat-8.6.0-2023.01.17-000001 | 704,062 (51.3%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `604` **ECS compliant fields** `1209` **All fields** `1813`
### .ds-packetbeat-8.4.1-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-packetbeat-8.4.1-2023.01.24-000001 | 668,831 (48.7%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `604` **ECS compliant fields** `1209` **All fields** `1813`
## Errors
Some indices were not checked for Data Quality
Errors may occur when pattern or index metadata is temporarily unavailable, or because you don't have the privileges required for access
The following privileges are required to check an index:
- `monitor` or `manage`
- `view_index_metadata`
- `read` or `read_cross_cluster`
| Pattern | Index | Error |
|---------|-------|-------|
| .alerts-security.alerts-default | -- | `Error loading stats: Error: Forbidden` |
| auditbeat-* | -- | `Error loading stats: Error: Forbidden` |
| logs-* | -- | `Error loading stats: Error: Forbidden` |
| packetbeat-* | -- | `Error loading stats: Error: Forbidden` |
See also: https://github.com/elastic/security-team/issues/4559
## Summary
Fix https://github.com/elastic/kibana/issues/148412
More and more SO types will not be accessible from the HTTP APIs (either
`hidden:true` or `hiddenFromHTTPApis: true`).
However, the FTR SO client (`KbnClientSavedObjects`) still needs to be
able to access and manipulate all SO types.
This PR introduces a `ftrSoApis` plugin that is loaded for all FTR
suites. This plugin exposes SO APIs that are used by the FTR client
instead of the public SO HTTP APIs. These APIs are configured to know
about all types, even hidden ones.
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Dearest Reviewers 👋
I've been working on this branch with @mistic and @tylersmalley and
we're really confident in these changes. Additionally, this changes code
in nearly every package in the repo so we don't plan to wait for reviews
to get in before merging this. If you'd like to have a concern
addressed, please feel free to leave a review, but assuming that nobody
raises a blocker in the next 24 hours we plan to merge this EOD pacific
tomorrow, 12/22.
We'll be paying close attention to any issues this causes after merging
and work on getting those fixed ASAP. 🚀
---
The operations team is not confident that we'll have the time to achieve
what we originally set out to accomplish by moving to Bazel with the
time and resources we have available. We have also bought ourselves some
headroom with improvements to babel-register, optimizer caching, and
typescript project structure.
In order to make sure we deliver packages as quickly as possible (many
teams really want them), with a usable and familiar developer
experience, this PR removes Bazel for building packages in favor of
using the same JIT transpilation we use for plugins.
Additionally, packages now use `kbn_references` (again, just copying the
dx from plugins to packages).
Because of the complex relationships between packages/plugins and in
order to prepare ourselves for automatic dependency detection tools we
plan to use in the future, this PR also introduces a "TS Project Linter"
which will validate that every tsconfig.json file meets a few
requirements:
1. the chain of base config files extended by each config includes
`tsconfig.base.json` and not `tsconfig.json`
1. the `include` config is used, and not `files`
2. the `exclude` config includes `target/**/*`
3. the `outDir` compiler option is specified as `target/types`
1. none of these compiler options are specified: `declaration`,
`declarationMap`, `emitDeclarationOnly`, `skipLibCheck`, `target`,
`paths`
4. all references to other packages/plugins use their pkg id, ie:
```js
// valid
{
"kbn_references": ["@kbn/core"]
}
// not valid
{
"kbn_references": [{ "path": "../../../src/core/tsconfig.json" }]
}
```
5. only packages/plugins which are imported somewhere in the ts code are
listed in `kbn_references`
This linter is not only validating all of the tsconfig.json files, but
it also will fix these config files to deal with just about any
violation that can be produced. Just run `node scripts/ts_project_linter
--fix` locally to apply these fixes, or let CI take care of
automatically fixing things and pushing the changes to your PR.
> **Example:** [`64e93e5`
(#146212)](64e93e5806)
When I merged main into my PR it included a change which removed the
`@kbn/core-injected-metadata-browser` package. After resolving the
conflicts I missed a few tsconfig files which included references to the
now removed package. The TS Project Linter identified that these
references were removed from the code and pushed a change to the PR to
remove them from the tsconfig.json files.
## No bazel? Does that mean no packages??
Nope! We're still doing packages but we're pretty sure now that we won't
be using Bazel to accomplish the 'distributed caching' and 'change-based
tasks' portions of the packages project.
This PR actually makes packages much easier to work with and will be
followed up with the bundling benefits described by the original
packages RFC. Then we'll work on documentation and advocacy for using
packages for any and all new code.
We're pretty confident that implementing distributed caching and
change-based tasks will be necessary in the future, but because of
recent improvements in the repo we think we can live without them for
**at least** a year.
## Wait, there are still BUILD.bazel files in the repo
Yes, there are still three webpack bundles which are built by Bazel: the
`@kbn/ui-shared-deps-npm` DLL, `@kbn/ui-shared-deps-src` externals, and
the `@kbn/monaco` workers. These three webpack bundles are still created
during bootstrap and remotely cached using bazel. The next phase of this
project is to figure out how to get the package bundling features
described in the RFC with the current optimizer, and we expect these
bundles to go away then. Until then any package that is used in those
three bundles still needs to have a BUILD.bazel file so that they can be
referenced by the remaining webpack builds.
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Files management UI that rounds out the files MVP. This is UI is
intended to be progressively enhanced and provides a way for system
administrators get some insight and manage the files created and stored
in Kibana.
## To reviewers
* This is UI for retrieval and deletion of files (the R+D of CRUD)
* Creating and deleting tags to be supported in a future version
* This UI is intended to form part of the broader content management
experience
* We use the `TableListView` component as far as possible
## How to test
1. Start Kibana with `yarn start --run-examples`
2. Go to the "Developer Examples" from the left nav menu
3. Go to the "Files example" plugin
4. Click the "Upload file" button, upload a few different image types
(PNG, JPG and WEBP)
5. Go to "Stack management" > "Files"
6. Behold your files in the management UI
7. (Bonus) check that the UI and API `GET /api/files/find`, `GET
/api/files/metrics` and `DELETE /api/files/blobs` are not accessible to
non-admin or appropriately privileged users (i.e., those with "Files
management" access).
## List of functionality
- [x] List all saved objects (scoped to admin)
- [x] Is able to bulk-delete files
- [x] Shows basic storage diagnostics
- [x] Is able to search and filter files
## Screenshots
<details>
<summary>screenshots</summary>
<img width="1545" alt="Screenshot 2022-11-08 at 13 56 54"
src="https://user-images.githubusercontent.com/8155004/200570783-cfefdbf3-c5ff-4ece-ba24-48a455fcca75.png">
<img width="910" alt="Screenshot 2022-11-10 at 12 52 35"
src="https://user-images.githubusercontent.com/8155004/201083812-bc9f25f5-b423-43a6-9229-5e2a4cdd943a.png">
<img width="451" alt="Screenshot 2022-11-10 at 12 37 07"
src="https://user-images.githubusercontent.com/8155004/201081039-832a1980-684c-4abb-bb05-0c7c6a849d4d.png">
<img width="959" alt="Screenshot 2022-11-08 at 13 57 15"
src="https://user-images.githubusercontent.com/8155004/200570797-f122cff5-7043-4e01-9b51-d5663c1b26d6.png">
<img width="500" alt="Screenshot 2022-11-08 at 13 57 38"
src="https://user-images.githubusercontent.com/8155004/200570801-35cdbd99-0256-4dee-9f78-2f6ad853305f.png">
</details>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
* Makes the API docs visible on dev site
* Updates the file README
* Updates the file service tutorial (more structured steps, leverage our
own UI components)
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* [Discover] Create unifiedHistogram plugin
* [Discover] Move discover resizable panels to unifiedHistogram
* [Discover] Replace DiscoverPanels with unifiedHistogram Panels
* [CI] Auto-commit changed files from 'node scripts/build_plugin_list_docs'
* [Discover] Fix types and limtis.yml for unifiedHistogram
* [Discover] Begin migrating layout and chart to unified_histogram
* [Discover] Update i18n keys from discover to unifiedHistogram
* [Discover] Update data-test-subj tags from discover to unifiedHistogram
* [Discover] Update classNames, ids, and scss to change discover to unifiedHistogram
* [Discover] Remove more references to discover from unifiedHistogram
* [Discover] Replace DiscoverServices with UnifiedHistogramServices
* [Discover] Replacing CHART_HIDDEN_KEY with chartHiddenKey prop
* [Discover] Add missing tsconfig references
* [Discover] Remove remaining references to discover from unifiedHistogram
* [Discover] Migrate HitsCounter to unifiedHistogram
* [Discover] Continue removing discover dependencies from unifiedHistogram
* [Discover] Replace SCSS with emotion
* [Discover] Changing PANELS_MODE to be internal
* [Discover] Clean up types
* [Discover] Clean up props and types
* [Discover] Update layout to use Chart component
* [Discover] Update discover_main_content
* [CI] Auto-commit changed files from 'node scripts/precommit_hook.js --ref HEAD~1..HEAD --fix'
* [Discover] Update discover_main_content to use UnifiedHistogramLayout, clean up unifiedHistogram implementation and props, add missing bundles
* [Discover] Fix missing styles in unifiedHistogram
* [Discover] Fix issue where mouse can get out of sync with the resize handle with the Discover resizable layout
* [Discover] Fix some Jest tests
* [Discover] Update discoverQueryHits to unifiedHistogramQueryHits in tests
* [Discover] Finish decoupling discover_main_content from unified histogram layout
* [Discover] Create useDiscoverHistogram hook and remove old histogram dependencies from Discover
* [Discover] Move functions to create chart data from discover to unifiedHistogram
* [Discover] Continue fixing broken Jest tests
* Revert unifiedHistogram.reloadSavedSearchButton removal
* [Discover] Add missing type export and a better suspense fallback
* [Discover] Make callback names consistent
* [Discover] Continue cleanup and add documentation to unifiedHistogram
* [Discover] Update genChartAggConfigs to take object
* [Discover] Update UnifiedHistogramHitsContext.number to total
* [Discover] Cleanup imports
* [Discover] Add support for hiding the entire top panel in the unified histogram by leaving all context props undefined
* [Discover] Fix broken discover_layout unit tests
* [Discover] Clean up naming in discover_main_content
* [Discover] Continue fixing Jest tests and adding new tests
* [Discover] Finish writing Jest tests
* [Discover] Fix conflicts with getVisualizeInformation and triggerVisualizeActions after merge
* [Discover] Fix hiding reset chart height button when default chart height
* [Discover] Update CODEOWNER file
* [Discover] Removed types for @link comments
* [Discover] Fix broken discover_layout.test.tsx file
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* [Guided onboarding] Smashed commit of all POC work for guided onboarding and guided onboarding example plugins
* [Guided onboarding] Fixed type errors
* [Guided onboarding] Removed guidedOnboardingExample limit
* [Guided onboarding] Fixed a functonal test for exposed configs
* [Guided onboarding] Fixed plugin limit
* [Guided onboarding] Added more information to the example plugin
* [Guided onboarding] Fixed no-console error
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
* [Guided onboarding] Fixed snake case errors
* move guided_onboarding out of x-pack
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Alison Goryachev <alison.goryachev@elastic.co>
* [Discover] Add initial support for tags to saved search modal
* [Discover] Add tags to savedSearch types
* [Discover] Finish initial support for adding tags to saved searches
* [Discover] Start to convert saved object finder to a table in order to support tags
* [Discover] Add support for displaying saved search tags in open search flyout
* [CI] Auto-commit changed files from 'node scripts/precommit_hook.js --ref HEAD~1..HEAD --fix'
* [Discover] Continue support for tags in saved object finder
* [Discover] Clean up saved object finder
* [Discover] Finish initial support for tags in saved object finder
* [Discover] Update SimpleSavedObject constructor to SimpleSavedObjectImpl
* [Discover] Remove orig files
* [Discover] Saved search tag type registration and telemetry
* [Discover] Create new saved_objects_finder plugin
* [Discover] Continue work creating saved_objects_finder plugin
* [Discover] Revert some changes in saved_objects
* [Discover] Revert some changes in saved_objects again
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
* [Discover] Update saved_objects_finder i18n keys
* [Discover] Update docs
* [Discover] Add plugins to saved_object_finder and fix broken types
* [Discover] Finish creating saved_objects_finder plugin and use it in Discover
* [Discover] Update SavedObjectFinderProps type export, and update x-pack telemetry
* [Discover] Fix broken jest tests
* [Discover] Update saved_objects_finder API
* [Discover] Remove unused translations
* [Discover] Fix issue with initial saved object finder fetch
* [Discover] Fix some of the saved object finder jest tests
* [Discover] Clean up finder after merge
* [Discover] Fixing saved_object_finder.tsx
* [Discover] Add savedObjectsTaggingOss reference to saved_search plugin
* [Discover] Fix broken open_search_panel test
* [Discover] Removed allowed types from saved object finder
* [Discover] Removing type column when there's only one saved object type, and adjusting column widths
* [Discover] Fix issue where visible types were entirely removed, fixed pageSizeOptions
* [Discover] Add showFilter to open_search_panel's saved_objects_finder, fallback to all available types when no type filter is applied to saved_objects_finder, hide type column and filter button when there is only one type in metadata list
* [Discover] Fix remaining saved_object_finder Jest tests
* [Discover] Update snapshot
* [Discover] Fix failing functional tests
* [Discover] Add tagging Jest tests for saved_objects_finder
* [Discover] Fix small bugs in saved_object_finder, update Jest tests, add functional tests for Discover saved search tagging
* [Discover] Removed unused variable in functional test
* [Discover] Update Discover Jest tests with tagging tests
* [Discover] Remove translations
* [Discover] Updating saved_objects_finder to use static export vs preconfigured component, adding lazy load support
* [Discover] Move saved_object_finder service deps to a 'services' prop, fix broken open_search_panel Jest test
* [Discover] Fix broken Jest test
* [Discover] Fix broken Jest test from merge
* [Discover] Fix discover tags integration test description
* - Updated tags prop to be `string | undefined`
- Type imports cleanup
- Added loading indicator for saved object finder
- Changed `savedObjectsPlugin.settings.getListingLimit()` to `uiSettings.get(LISTING_LIMIT_SETTING)`
- Removed old saved object finder comment
- Removed tag changes from transform plugin
- Change code owners of saved_objects_finder to Data Discovery
* [Discover] Fixed LISTING_LIMIT_SETTING import
* [Discover] Revert tags saving change that introduced a bug
* [CI] Auto-commit changed files from 'node scripts/build_plugin_list_docs'
* [Discover] Try again to fix LISTING_LIMIT_SETTINGS import
* [Discover] Fix failing snapshot
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* added files plugin
* [Files] Create files saved object and blob storage (#131886)
* initial version of file saved objects
* initial version of file saved objects part ii
* added .blob index and setup logic for the file service and the blob service
* wip: need to add .blob access to kibana_system role
* use multiple-isolated after speaking of oleg
* added metadata field
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
* [CI] Auto-commit changed files from 'node scripts/build_plugin_list_docs'
* updated file statuses per feedback
* removed created_by and added "alt" text field and converted "name" to type "text" too
* split content_type into mime and extension
* use BlobStorage interface in adapters prop
* minor fixes
* handle multiple Kibanas at startup time
* fix jest test
* do not set up index at setup time
* set importable exportable to false explicitly
* remove use of non-existent function
* added plugin ID, use type instead of interface to avoid specific interface issue and use typescript to check that properties map corresponds to type
* remove storage_id for now
* make the name snake case!
* added fixmes
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* [Files] Implement ES-based blob store (#132123)
* minor clean up
* remove unused import
* added modified version of the content stream from reporting
* remove plugin_id, replace with file_kind
* remove unused commented out code and clean up imports
* added jest.config.js
* added FileChunkDocument interface and added the head_chunk_id for easier deletion of related file chunks
* added an additional test case for writing with content stream
* first implementation of ES blob storage
* fix imports
* get closer to a final version of the blobstore interface, use esClient.get rather than esClient.search since we are using IDs
* added jest integration tests
* add some comments
* fix: eslint types issues
* remove unused values
* minor refactor for clarity
* fix when saved objects are registered
* remove unused variable
* fix jest tests and make getMaxSize function sync
* fix logger prefixes
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [Files] File service CRUD functionality and audit logging (#132416)
* big ol wipperooni, but got a lot of initial functionality written out
* added some todos and fixed import
* finish adding integration tests for file service! also added the partial update capability to file
* added file service integration tests and finalized Files and InternalFileService APIs
* added security plugin
* big refactor for security audit logging: FileServiceFactory
* type lint
* update delete file integration test and remove done TODOs
* move comment
* docs, accessibility modifiers and updated some import ordering
* fix comments
* pass in index name to ES blob store
* tidy up debug and error messages
* [Files] Added more chunking Jest integration tests (#132530)
* added more thorough integration tests of chunking
* refactor chunksize to a value that is passed to the ES constructor, update a comment and update tests
* [Files] Public API v1 (#132602)
* created public API, everything except the file registry
* ensure the custom meta types are working as expected
* added index: false to content
* Revert "added index: false to content"
This reverts commit 74259f86638aefc24755874cbd6ac89e0170c192.
* updated comment
* update how ids are generated
* also filter out deleted files
* fix id generation
* test file deletion after upload failure
* [Files] File kinds registry v0.1 (#132700)
* initial commit of file kind registry
* add actual registry
* updated attributes reduceer slightly
* separate blob storage interface, make the blob storage type a unique const
* added integration test for file kinds
* update test after adding registry logic
* partial attributes
* [Files] Blob storage with attributes (#134057)
* updated blob storage interface with attributes
* added dynamic: false to the file chunk doc for ES blob store
* update mocks
* support attributes at the blob store level part i
* actually test setting of attributes, also refactor app_extra_data to app_meta_data
* remove unused import
* some corrections to tests, also refresh after creating a blob that has attributes to enable search
* remove unnecessary default
* getBaseId -> getContentReferenceId and added a bunch of doc comments
* refactored getting of attribute chunk id, encapsulate everything inside content stream
* remove app_search_data
* app_meta_data -> app_metadata
* [Files] Added `Transform` to blob stores (#134846)
* added max byte size stream transform
* added stream transforms to files
* rename MaxByteSize to MaxByteSizeTransform
* try a cool cool ternary, yeah
* minor refactor
* address PR feedback
* [Files] File kind HTTP APIs and tests (#134655)
* first version of dynamically creating file kinds routes
* added update, delete, download, list and find endpoints and some minor refactoring
* added return statement and additional requirement for file kind ids to be URI safe
* update comment
* added API integration tests for the file kinds routes, a lot of refactoring; removed the upload endpoint service since this does not make since with proxies, fixed a bunch of issues in registering routes and the enhanced file kind router
* move the method to the specific route file too
* remove unused function
* add download content type response header
* remove unused import
* download http method should be "get" not "delete"
* actually pass through the mime type
* slight refactor after changing the create file response
* findFile => getById
* remove trailing verbs
* update copy
* addressing pr feedback
* added missing property
* fix copy
* added comment
* regrouping imoprts
* rename "find" route to "getById"
* [File storage] Public-side Client (#135403)
* refactor store name
* wip: first version of restful-ish endpoint types captured in one place except for path
* use the endpoint definition directly
* added first iteration of files client based on shared types
* slight refactor
* added content type headers and removed unused type
* slight refactor and expose server and public side types
* finish refactor
* updated files plugin ui limits
* [Files] Public file sharing service (#135598)
* add new public file saved object type
* generate random token
* file share service, first iteration
* initial version of files share service work, we can create a file share
* delete shares when a file is deleted
* refactor to InternalFileSharingService because certain events need to be audited
* further clean up and added a comment
* finish refactor
* update tests
* refactor tractor
* move integration tests folder
* does not need to be async, also added some samples to doc comment
* added comment
* use SO object references instead of hacking it yourself, also added comments and moved some stuff around
* refactor to internal file share service
* fix type lint
* valid_until as unix ts
* commit simplify random string generator
* fix type issues and update SO integration tests
* woops, 51 not 55
* change label
* minor refactor to file service (#135836)
* remove attributes from blobstore layer and update IDs to be xxxxx.1 instead of 1.xxxxx (#136294)
* [Files] Refactor File metadata (#136370)
* refactor tractor keeps on rolling
* fix type issues
* [Files] Conform with new blob store specification (#136396)
* refactor es mappings
* move puid file
* updated the content stream implementation to support identifying "last" chunk
* updated the es blob store
* updated es mappings
* minor updates to the file object, but importantly, passing the file ID to the blob store so that files and blobs are connected
* updated test assertion
* rather use cuid because it uses cryptography to reduce the chance of collisions (#136658)
* do not index bid field (#136707)
* [Files] Find endpoint (#136529)
* refactor http endpoint interface names
* wip on find files route, need to create shared test harness and test utils
* added toJSON helper
* pass through the filter
* fix refactor to "getById" on file service instead of find and move order of attributes
* actually add the toJSON utility
* added shared integration test setup code
* added find route path to common
* return if 400 is returned when trying to create blob storage index. This is an edge case that can be hit if creation happens at the same time
* update the find algo to build a kuery expression string that we pass to SO service, note: we want to prevent query injection shenanigans so we JSON.stringify
* updated the create endpoint to enforce mime type from file type, thank you
* updated the create endpoint to enforce mime type from file type, thank you
* updated some tests and actually register the find enpoint, also updated to use mimeType
* added a few integration level tests for the new find endpoint
* updated the new setup integration test harness for use in new find and old file kinds integration tests
* use flattened type for "Meta" and "hash" fields on saved object so that we can store multiple values in an object and search
* updated import with "type" annotation
* added readme explaining patterns used for routes
* fix file kind integration tests
* remove unused endpoint
* do not expose chunk size and compression for now
* use nodeBuilder rather than manually building a kuery string
* make query strings limited in length and refactor "Extension" to "extension" plus some other minor changes
* reorg some lines
* [Files] Use `application/cbor` to optimise file uploads and downloads (#136528)
* move puid file
* added cbor-x dependency
* remove all references to "raw" from the test code
* remove "encoding" parameter and refactor to use cbor encoding for blob chunks, in this way we do not encoding our payloads to UTF-8 or Base64 Kibana side, only decode
* fixed content stream tests
* require instead of import cbor-x for now
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
* try disable eslint comment scoped
* download also as cbor to buffer directly, also updated upload endpoint to report when content has already been uploaded
* updated tests and handle some edge case when reading buffers
* remove unused require and added a comment
* moved lengthy comment to separate file
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* [Files] Added metrics endpoint and functionality (#136725)
* added service-level functionality for getting usage metrics
* built the metrics endpoint
* do not count size of deleted files
* updated comment
* added agg by status and extension
* update test assertions
* [Files] File errors (#136946)
* added domain-specific errors
* map file errors to http responses and clean up list endpoint
* [Files] Public File share HTTP endpoints (#137172)
* refactor where server and client routes are declared
* added new endpoint types and refactor api/files/files/metrics|share to api/files/metrics|share
* added some more comments, a new error type for the file share service and created the endpoint handler for creating file shares
* added doc comments to the file kind interface
* rename test utils folder to "test_utils" and added a basic test for creating a share
* added test_util folder and updated share test
* added new endpoint types, updated import to test utils, updated share name
* added unshare endpoint
* added share list endpoint and refactored share JSON with version that contains token and version without
* actually register the list shares route
* a bunch of refactoring for saved object type "id" rather than "token" for the object id, also added the public download endpoint
* updated error behavior on download endpoint
* fixed test
* use unix timestamp in seconds
* added file name param to public download endpoint and added shared fileName schema
* remove unused file and added public download tests
* share to shares
* added get endopint and updated tests and test titles per feedback
* added get endpoint
* updated test and back to milliseconds on validUntil
* updated name of params in route and update error message in response
* return only known messages in the expected format
* update test
* use Readable.from
* reintroduce crazy typecasting
* added security audit log tests (#137253)
* [Files] Generate download headers utility tests (#137203)
* move common schema to common schema file and move THAT file to a shared location
* update mime type behaviour
* update integration tests
* added a little something to the README
* update cbor-x to latest
* delete "require" of cbor-x
* upgrade cbor-x
* rename "public" to "public_facing"
* remove old lint rule
* added a bunch of docs
* fix interface surface area
* more doc comments
* more more doc comments
* more more more doc comments
* more more more more doc comments
* more more more more more more doc comments
* Actually 50GiB
Co-authored-by: Vadim Kibana <82822460+vadimkibana@users.noreply.github.com>
* Remove unnecessary comment
Co-authored-by: Vadim Kibana <82822460+vadimkibana@users.noreply.github.com>
* hasContent -> isReady
* only call the createIndexIfNotExists function once per instance
* [Files] Create files client (#137879)
* biig refactor to use FileClient in File and remove all metadata functionality from file service
* update file service factory to work with metadata client after refactor
* also rename file share updateable attrs interface
* updated use of types in saved objects client
* export function arg types
* added the creation helper file and added an example file for using it
* implement find for the es index client
* moved the query builder logic to a shared place
* rename file, add prefixes and embed es doc in "file" key in ES document
* type exports
* fix bad rebase
* Minor changes and updates to types
* fix another minor type issue
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
* fix minor type issues and update "update" behavior to only apply updatable fields bru
* [CI] Auto-commit changed files from 'node scripts/precommit_hook.js --ref HEAD~1..HEAD --fix'
* actually be able to update status... also some other types cleanups
* [CI] Auto-commit changed files from 'node scripts/precommit_hook.js --ref HEAD~1..HEAD --fix'
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* remove TODO
* update link to team
* throw errors more consistently
* createAuditLog -> writeAuditLog
* more more more more more more more doc comments
* more more more more more more more more doc comments
* more more more more more more more more more doc comments
* more more more more more more more more more more doc comments
* more more more more more more more more more more more more doc comments
* [Files] ES client tests (#138174)
* move test utils and update expected metric size
* remove cuid from "file"
* update file client to set id using cuid and update types
* moved test utils and wip of integration tests for file client
* added es-index backed file client tests
* adjust the upload endpoint to also return the file size
* added size to response expect
* fix some type issues
* fix type issues
* [Files] Register routes at setup time (#138392)
* updated the fileKindRegistory to live behind a getter setter so that instantation can be taken care of in one place
* refactor conditional endpoint registration and update tests to register file kind after setup
* register routes at setup time
* remove outdated README
* fix import of non-existent function
* register routes...
* [Files] PR feedback 1 (#138417)
* update comments and plugin description
* filter lists and finds on non-deleted files
* updated comment
* added tests for larger files and files exactly divisible by chunk size
* [CI] Auto-commit changed files from 'node scripts/build_plugin_list_docs'
* do not use regexp
* fix test assertion
* update content stream tests
* make fewer fields searchable
* ok
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* remove legacy comment
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Vadim Kibana <82822460+vadimkibana@users.noreply.github.com>
