mirror of
https://github.com/elastic/kibana.git
synced 2025-04-25 02:09:32 -04:00
a182b8e9be
* added files plugin * [Files] Create files saved object and blob storage (#131886) * initial version of file saved objects * initial version of file saved objects part ii * added .blob index and setup logic for the file service and the blob service * wip: need to add .blob access to kibana_system role * use multiple-isolated after speaking of oleg * added metadata field * [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix' * [CI] Auto-commit changed files from 'node scripts/build_plugin_list_docs' * updated file statuses per feedback * removed created_by and added "alt" text field and converted "name" to type "text" too * split content_type into mime and extension * use BlobStorage interface in adapters prop * minor fixes * handle multiple Kibanas at startup time * fix jest test * do not set up index at setup time * set importable exportable to false explicitly * remove use of non-existent function * added plugin ID, use type instead of interface to avoid specific interface issue and use typescript to check that properties map corresponds to type * remove storage_id for now * make the name snake case! * added fixmes Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> * [Files] Implement ES-based blob store (#132123) * minor clean up * remove unused import * added modified version of the content stream from reporting * remove plugin_id, replace with file_kind * remove unused commented out code and clean up imports * added jest.config.js * added FileChunkDocument interface and added the head_chunk_id for easier deletion of related file chunks * added an additional test case for writing with content stream * first implementation of ES blob storage * fix imports * get closer to a final version of the blobstore interface, use esClient.get rather than esClient.search since we are using IDs * added jest integration tests * add some comments * fix: eslint types issues * remove unused values * minor refactor for clarity * fix when saved objects are registered * remove unused variable * fix jest tests and make getMaxSize function sync * fix logger prefixes Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> * [Files] File service CRUD functionality and audit logging (#132416) * big ol wipperooni, but got a lot of initial functionality written out * added some todos and fixed import * finish adding integration tests for file service! also added the partial update capability to file * added file service integration tests and finalized Files and InternalFileService APIs * added security plugin * big refactor for security audit logging: FileServiceFactory * type lint * update delete file integration test and remove done TODOs * move comment * docs, accessibility modifiers and updated some import ordering * fix comments * pass in index name to ES blob store * tidy up debug and error messages * [Files] Added more chunking Jest integration tests (#132530) * added more thorough integration tests of chunking * refactor chunksize to a value that is passed to the ES constructor, update a comment and update tests * [Files] Public API v1 (#132602) * created public API, everything except the file registry * ensure the custom meta types are working as expected * added index: false to content * Revert "added index: false to content" This reverts commit 74259f86638aefc24755874cbd6ac89e0170c192. * updated comment * update how ids are generated * also filter out deleted files * fix id generation * test file deletion after upload failure * [Files] File kinds registry v0.1 (#132700) * initial commit of file kind registry * add actual registry * updated attributes reduceer slightly * separate blob storage interface, make the blob storage type a unique const * added integration test for file kinds * update test after adding registry logic * partial attributes * [Files] Blob storage with attributes (#134057) * updated blob storage interface with attributes * added dynamic: false to the file chunk doc for ES blob store * update mocks * support attributes at the blob store level part i * actually test setting of attributes, also refactor app_extra_data to app_meta_data * remove unused import * some corrections to tests, also refresh after creating a blob that has attributes to enable search * remove unnecessary default * getBaseId -> getContentReferenceId and added a bunch of doc comments * refactored getting of attribute chunk id, encapsulate everything inside content stream * remove app_search_data * app_meta_data -> app_metadata * [Files] Added `Transform` to blob stores (#134846) * added max byte size stream transform * added stream transforms to files * rename MaxByteSize to MaxByteSizeTransform * try a cool cool ternary, yeah * minor refactor * address PR feedback * [Files] File kind HTTP APIs and tests (#134655) * first version of dynamically creating file kinds routes * added update, delete, download, list and find endpoints and some minor refactoring * added return statement and additional requirement for file kind ids to be URI safe * update comment * added API integration tests for the file kinds routes, a lot of refactoring; removed the upload endpoint service since this does not make since with proxies, fixed a bunch of issues in registering routes and the enhanced file kind router * move the method to the specific route file too * remove unused function * add download content type response header * remove unused import * download http method should be "get" not "delete" * actually pass through the mime type * slight refactor after changing the create file response * findFile => getById * remove trailing verbs * update copy * addressing pr feedback * added missing property * fix copy * added comment * regrouping imoprts * rename "find" route to "getById" * [File storage] Public-side Client (#135403) * refactor store name * wip: first version of restful-ish endpoint types captured in one place except for path * use the endpoint definition directly * added first iteration of files client based on shared types * slight refactor * added content type headers and removed unused type * slight refactor and expose server and public side types * finish refactor * updated files plugin ui limits * [Files] Public file sharing service (#135598) * add new public file saved object type * generate random token * file share service, first iteration * initial version of files share service work, we can create a file share * delete shares when a file is deleted * refactor to InternalFileSharingService because certain events need to be audited * further clean up and added a comment * finish refactor * update tests * refactor tractor * move integration tests folder * does not need to be async, also added some samples to doc comment * added comment * use SO object references instead of hacking it yourself, also added comments and moved some stuff around * refactor to internal file share service * fix type lint * valid_until as unix ts * commit simplify random string generator * fix type issues and update SO integration tests * woops, 51 not 55 * change label * minor refactor to file service (#135836) * remove attributes from blobstore layer and update IDs to be xxxxx.1 instead of 1.xxxxx (#136294) * [Files] Refactor File metadata (#136370) * refactor tractor keeps on rolling * fix type issues * [Files] Conform with new blob store specification (#136396) * refactor es mappings * move puid file * updated the content stream implementation to support identifying "last" chunk * updated the es blob store * updated es mappings * minor updates to the file object, but importantly, passing the file ID to the blob store so that files and blobs are connected * updated test assertion * rather use cuid because it uses cryptography to reduce the chance of collisions (#136658) * do not index bid field (#136707) * [Files] Find endpoint (#136529) * refactor http endpoint interface names * wip on find files route, need to create shared test harness and test utils * added toJSON helper * pass through the filter * fix refactor to "getById" on file service instead of find and move order of attributes * actually add the toJSON utility * added shared integration test setup code * added find route path to common * return if 400 is returned when trying to create blob storage index. This is an edge case that can be hit if creation happens at the same time * update the find algo to build a kuery expression string that we pass to SO service, note: we want to prevent query injection shenanigans so we JSON.stringify * updated the create endpoint to enforce mime type from file type, thank you * updated the create endpoint to enforce mime type from file type, thank you * updated some tests and actually register the find enpoint, also updated to use mimeType * added a few integration level tests for the new find endpoint * updated the new setup integration test harness for use in new find and old file kinds integration tests * use flattened type for "Meta" and "hash" fields on saved object so that we can store multiple values in an object and search * updated import with "type" annotation * added readme explaining patterns used for routes * fix file kind integration tests * remove unused endpoint * do not expose chunk size and compression for now * use nodeBuilder rather than manually building a kuery string * make query strings limited in length and refactor "Extension" to "extension" plus some other minor changes * reorg some lines * [Files] Use `application/cbor` to optimise file uploads and downloads (#136528) * move puid file * added cbor-x dependency * remove all references to "raw" from the test code * remove "encoding" parameter and refactor to use cbor encoding for blob chunks, in this way we do not encoding our payloads to UTF-8 or Base64 Kibana side, only decode * fixed content stream tests * require instead of import cbor-x for now * [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix' * try disable eslint comment scoped * download also as cbor to buffer directly, also updated upload endpoint to report when content has already been uploaded * updated tests and handle some edge case when reading buffers * remove unused require and added a comment * moved lengthy comment to separate file Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> * [Files] Added metrics endpoint and functionality (#136725) * added service-level functionality for getting usage metrics * built the metrics endpoint * do not count size of deleted files * updated comment * added agg by status and extension * update test assertions * [Files] File errors (#136946) * added domain-specific errors * map file errors to http responses and clean up list endpoint * [Files] Public File share HTTP endpoints (#137172) * refactor where server and client routes are declared * added new endpoint types and refactor api/files/files/metrics|share to api/files/metrics|share * added some more comments, a new error type for the file share service and created the endpoint handler for creating file shares * added doc comments to the file kind interface * rename test utils folder to "test_utils" and added a basic test for creating a share * added test_util folder and updated share test * added new endpoint types, updated import to test utils, updated share name * added unshare endpoint * added share list endpoint and refactored share JSON with version that contains token and version without * actually register the list shares route * a bunch of refactoring for saved object type "id" rather than "token" for the object id, also added the public download endpoint * updated error behavior on download endpoint * fixed test * use unix timestamp in seconds * added file name param to public download endpoint and added shared fileName schema * remove unused file and added public download tests * share to shares * added get endopint and updated tests and test titles per feedback * added get endpoint * updated test and back to milliseconds on validUntil * updated name of params in route and update error message in response * return only known messages in the expected format * update test * use Readable.from * reintroduce crazy typecasting * added security audit log tests (#137253) * [Files] Generate download headers utility tests (#137203) * move common schema to common schema file and move THAT file to a shared location * update mime type behaviour * update integration tests * added a little something to the README * update cbor-x to latest * delete "require" of cbor-x * upgrade cbor-x * rename "public" to "public_facing" * remove old lint rule * added a bunch of docs * fix interface surface area * more doc comments * more more doc comments * more more more doc comments * more more more more doc comments * more more more more more more doc comments * Actually 50GiB Co-authored-by: Vadim Kibana <82822460+vadimkibana@users.noreply.github.com> * Remove unnecessary comment Co-authored-by: Vadim Kibana <82822460+vadimkibana@users.noreply.github.com> * hasContent -> isReady * only call the createIndexIfNotExists function once per instance * [Files] Create files client (#137879) * biig refactor to use FileClient in File and remove all metadata functionality from file service * update file service factory to work with metadata client after refactor * also rename file share updateable attrs interface * updated use of types in saved objects client * export function arg types * added the creation helper file and added an example file for using it * implement find for the es index client * moved the query builder logic to a shared place * rename file, add prefixes and embed es doc in "file" key in ES document * type exports * fix bad rebase * Minor changes and updates to types * fix another minor type issue * [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix' * fix minor type issues and update "update" behavior to only apply updatable fields bru * [CI] Auto-commit changed files from 'node scripts/precommit_hook.js --ref HEAD~1..HEAD --fix' * actually be able to update status... also some other types cleanups * [CI] Auto-commit changed files from 'node scripts/precommit_hook.js --ref HEAD~1..HEAD --fix' Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> * remove TODO * update link to team * throw errors more consistently * createAuditLog -> writeAuditLog * more more more more more more more doc comments * more more more more more more more more doc comments * more more more more more more more more more doc comments * more more more more more more more more more more doc comments * more more more more more more more more more more more more doc comments * [Files] ES client tests (#138174) * move test utils and update expected metric size * remove cuid from "file" * update file client to set id using cuid and update types * moved test utils and wip of integration tests for file client * added es-index backed file client tests * adjust the upload endpoint to also return the file size * added size to response expect * fix some type issues * fix type issues * [Files] Register routes at setup time (#138392) * updated the fileKindRegistory to live behind a getter setter so that instantation can be taken care of in one place * refactor conditional endpoint registration and update tests to register file kind after setup * register routes at setup time * remove outdated README * fix import of non-existent function * register routes... * [Files] PR feedback 1 (#138417) * update comments and plugin description * filter lists and finds on non-deleted files * updated comment * added tests for larger files and files exactly divisible by chunk size * [CI] Auto-commit changed files from 'node scripts/build_plugin_list_docs' * do not use regexp * fix test assertion * update content stream tests * make fewer fields searchable * ok Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> * remove legacy comment Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Vadim Kibana <82822460+vadimkibana@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| advanced | ||
| architecture | ||
| best-practices | ||
| contributing | ||
| getting-started | ||
| images | ||
| plugin | ||
| index.asciidoc | ||
| plugin-list.asciidoc | ||
| telemetry.asciidoc | ||