* added files plugin
* [Files] Create files saved object and blob storage (#131886)
* initial version of file saved objects
* initial version of file saved objects part ii
* added .blob index and setup logic for the file service and the blob service
* wip: need to add .blob access to kibana_system role
* use multiple-isolated after speaking of oleg
* added metadata field
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
* [CI] Auto-commit changed files from 'node scripts/build_plugin_list_docs'
* updated file statuses per feedback
* removed created_by and added "alt" text field and converted "name" to type "text" too
* split content_type into mime and extension
* use BlobStorage interface in adapters prop
* minor fixes
* handle multiple Kibanas at startup time
* fix jest test
* do not set up index at setup time
* set importable exportable to false explicitly
* remove use of non-existent function
* added plugin ID, use type instead of interface to avoid specific interface issue and use typescript to check that properties map corresponds to type
* remove storage_id for now
* make the name snake case!
* added fixmes
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* [Files] Implement ES-based blob store (#132123)
* minor clean up
* remove unused import
* added modified version of the content stream from reporting
* remove plugin_id, replace with file_kind
* remove unused commented out code and clean up imports
* added jest.config.js
* added FileChunkDocument interface and added the head_chunk_id for easier deletion of related file chunks
* added an additional test case for writing with content stream
* first implementation of ES blob storage
* fix imports
* get closer to a final version of the blobstore interface, use esClient.get rather than esClient.search since we are using IDs
* added jest integration tests
* add some comments
* fix: eslint types issues
* remove unused values
* minor refactor for clarity
* fix when saved objects are registered
* remove unused variable
* fix jest tests and make getMaxSize function sync
* fix logger prefixes
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [Files] File service CRUD functionality and audit logging (#132416)
* big ol wipperooni, but got a lot of initial functionality written out
* added some todos and fixed import
* finish adding integration tests for file service! also added the partial update capability to file
* added file service integration tests and finalized Files and InternalFileService APIs
* added security plugin
* big refactor for security audit logging: FileServiceFactory
* type lint
* update delete file integration test and remove done TODOs
* move comment
* docs, accessibility modifiers and updated some import ordering
* fix comments
* pass in index name to ES blob store
* tidy up debug and error messages
* [Files] Added more chunking Jest integration tests (#132530)
* added more thorough integration tests of chunking
* refactor chunksize to a value that is passed to the ES constructor, update a comment and update tests
* [Files] Public API v1 (#132602)
* created public API, everything except the file registry
* ensure the custom meta types are working as expected
* added index: false to content
* Revert "added index: false to content"
This reverts commit 74259f86638aefc24755874cbd6ac89e0170c192.
* updated comment
* update how ids are generated
* also filter out deleted files
* fix id generation
* test file deletion after upload failure
* [Files] File kinds registry v0.1 (#132700)
* initial commit of file kind registry
* add actual registry
* updated attributes reduceer slightly
* separate blob storage interface, make the blob storage type a unique const
* added integration test for file kinds
* update test after adding registry logic
* partial attributes
* [Files] Blob storage with attributes (#134057)
* updated blob storage interface with attributes
* added dynamic: false to the file chunk doc for ES blob store
* update mocks
* support attributes at the blob store level part i
* actually test setting of attributes, also refactor app_extra_data to app_meta_data
* remove unused import
* some corrections to tests, also refresh after creating a blob that has attributes to enable search
* remove unnecessary default
* getBaseId -> getContentReferenceId and added a bunch of doc comments
* refactored getting of attribute chunk id, encapsulate everything inside content stream
* remove app_search_data
* app_meta_data -> app_metadata
* [Files] Added `Transform` to blob stores (#134846)
* added max byte size stream transform
* added stream transforms to files
* rename MaxByteSize to MaxByteSizeTransform
* try a cool cool ternary, yeah
* minor refactor
* address PR feedback
* [Files] File kind HTTP APIs and tests (#134655)
* first version of dynamically creating file kinds routes
* added update, delete, download, list and find endpoints and some minor refactoring
* added return statement and additional requirement for file kind ids to be URI safe
* update comment
* added API integration tests for the file kinds routes, a lot of refactoring; removed the upload endpoint service since this does not make since with proxies, fixed a bunch of issues in registering routes and the enhanced file kind router
* move the method to the specific route file too
* remove unused function
* add download content type response header
* remove unused import
* download http method should be "get" not "delete"
* actually pass through the mime type
* slight refactor after changing the create file response
* findFile => getById
* remove trailing verbs
* update copy
* addressing pr feedback
* added missing property
* fix copy
* added comment
* regrouping imoprts
* rename "find" route to "getById"
* [File storage] Public-side Client (#135403)
* refactor store name
* wip: first version of restful-ish endpoint types captured in one place except for path
* use the endpoint definition directly
* added first iteration of files client based on shared types
* slight refactor
* added content type headers and removed unused type
* slight refactor and expose server and public side types
* finish refactor
* updated files plugin ui limits
* [Files] Public file sharing service (#135598)
* add new public file saved object type
* generate random token
* file share service, first iteration
* initial version of files share service work, we can create a file share
* delete shares when a file is deleted
* refactor to InternalFileSharingService because certain events need to be audited
* further clean up and added a comment
* finish refactor
* update tests
* refactor tractor
* move integration tests folder
* does not need to be async, also added some samples to doc comment
* added comment
* use SO object references instead of hacking it yourself, also added comments and moved some stuff around
* refactor to internal file share service
* fix type lint
* valid_until as unix ts
* commit simplify random string generator
* fix type issues and update SO integration tests
* woops, 51 not 55
* change label
* minor refactor to file service (#135836)
* remove attributes from blobstore layer and update IDs to be xxxxx.1 instead of 1.xxxxx (#136294)
* [Files] Refactor File metadata (#136370)
* refactor tractor keeps on rolling
* fix type issues
* [Files] Conform with new blob store specification (#136396)
* refactor es mappings
* move puid file
* updated the content stream implementation to support identifying "last" chunk
* updated the es blob store
* updated es mappings
* minor updates to the file object, but importantly, passing the file ID to the blob store so that files and blobs are connected
* updated test assertion
* rather use cuid because it uses cryptography to reduce the chance of collisions (#136658)
* do not index bid field (#136707)
* [Files] Find endpoint (#136529)
* refactor http endpoint interface names
* wip on find files route, need to create shared test harness and test utils
* added toJSON helper
* pass through the filter
* fix refactor to "getById" on file service instead of find and move order of attributes
* actually add the toJSON utility
* added shared integration test setup code
* added find route path to common
* return if 400 is returned when trying to create blob storage index. This is an edge case that can be hit if creation happens at the same time
* update the find algo to build a kuery expression string that we pass to SO service, note: we want to prevent query injection shenanigans so we JSON.stringify
* updated the create endpoint to enforce mime type from file type, thank you
* updated the create endpoint to enforce mime type from file type, thank you
* updated some tests and actually register the find enpoint, also updated to use mimeType
* added a few integration level tests for the new find endpoint
* updated the new setup integration test harness for use in new find and old file kinds integration tests
* use flattened type for "Meta" and "hash" fields on saved object so that we can store multiple values in an object and search
* updated import with "type" annotation
* added readme explaining patterns used for routes
* fix file kind integration tests
* remove unused endpoint
* do not expose chunk size and compression for now
* use nodeBuilder rather than manually building a kuery string
* make query strings limited in length and refactor "Extension" to "extension" plus some other minor changes
* reorg some lines
* [Files] Use `application/cbor` to optimise file uploads and downloads (#136528)
* move puid file
* added cbor-x dependency
* remove all references to "raw" from the test code
* remove "encoding" parameter and refactor to use cbor encoding for blob chunks, in this way we do not encoding our payloads to UTF-8 or Base64 Kibana side, only decode
* fixed content stream tests
* require instead of import cbor-x for now
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
* try disable eslint comment scoped
* download also as cbor to buffer directly, also updated upload endpoint to report when content has already been uploaded
* updated tests and handle some edge case when reading buffers
* remove unused require and added a comment
* moved lengthy comment to separate file
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* [Files] Added metrics endpoint and functionality (#136725)
* added service-level functionality for getting usage metrics
* built the metrics endpoint
* do not count size of deleted files
* updated comment
* added agg by status and extension
* update test assertions
* [Files] File errors (#136946)
* added domain-specific errors
* map file errors to http responses and clean up list endpoint
* [Files] Public File share HTTP endpoints (#137172)
* refactor where server and client routes are declared
* added new endpoint types and refactor api/files/files/metrics|share to api/files/metrics|share
* added some more comments, a new error type for the file share service and created the endpoint handler for creating file shares
* added doc comments to the file kind interface
* rename test utils folder to "test_utils" and added a basic test for creating a share
* added test_util folder and updated share test
* added new endpoint types, updated import to test utils, updated share name
* added unshare endpoint
* added share list endpoint and refactored share JSON with version that contains token and version without
* actually register the list shares route
* a bunch of refactoring for saved object type "id" rather than "token" for the object id, also added the public download endpoint
* updated error behavior on download endpoint
* fixed test
* use unix timestamp in seconds
* added file name param to public download endpoint and added shared fileName schema
* remove unused file and added public download tests
* share to shares
* added get endopint and updated tests and test titles per feedback
* added get endpoint
* updated test and back to milliseconds on validUntil
* updated name of params in route and update error message in response
* return only known messages in the expected format
* update test
* use Readable.from
* reintroduce crazy typecasting
* added security audit log tests (#137253)
* [Files] Generate download headers utility tests (#137203)
* move common schema to common schema file and move THAT file to a shared location
* update mime type behaviour
* update integration tests
* added a little something to the README
* update cbor-x to latest
* delete "require" of cbor-x
* upgrade cbor-x
* rename "public" to "public_facing"
* remove old lint rule
* added a bunch of docs
* fix interface surface area
* more doc comments
* more more doc comments
* more more more doc comments
* more more more more doc comments
* more more more more more more doc comments
* Actually 50GiB
Co-authored-by: Vadim Kibana <82822460+vadimkibana@users.noreply.github.com>
* Remove unnecessary comment
Co-authored-by: Vadim Kibana <82822460+vadimkibana@users.noreply.github.com>
* hasContent -> isReady
* only call the createIndexIfNotExists function once per instance
* [Files] Create files client (#137879)
* biig refactor to use FileClient in File and remove all metadata functionality from file service
* update file service factory to work with metadata client after refactor
* also rename file share updateable attrs interface
* updated use of types in saved objects client
* export function arg types
* added the creation helper file and added an example file for using it
* implement find for the es index client
* moved the query builder logic to a shared place
* rename file, add prefixes and embed es doc in "file" key in ES document
* type exports
* fix bad rebase
* Minor changes and updates to types
* fix another minor type issue
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
* fix minor type issues and update "update" behavior to only apply updatable fields bru
* [CI] Auto-commit changed files from 'node scripts/precommit_hook.js --ref HEAD~1..HEAD --fix'
* actually be able to update status... also some other types cleanups
* [CI] Auto-commit changed files from 'node scripts/precommit_hook.js --ref HEAD~1..HEAD --fix'
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* remove TODO
* update link to team
* throw errors more consistently
* createAuditLog -> writeAuditLog
* more more more more more more more doc comments
* more more more more more more more more doc comments
* more more more more more more more more more doc comments
* more more more more more more more more more more doc comments
* more more more more more more more more more more more more doc comments
* [Files] ES client tests (#138174)
* move test utils and update expected metric size
* remove cuid from "file"
* update file client to set id using cuid and update types
* moved test utils and wip of integration tests for file client
* added es-index backed file client tests
* adjust the upload endpoint to also return the file size
* added size to response expect
* fix some type issues
* fix type issues
* [Files] Register routes at setup time (#138392)
* updated the fileKindRegistory to live behind a getter setter so that instantation can be taken care of in one place
* refactor conditional endpoint registration and update tests to register file kind after setup
* register routes at setup time
* remove outdated README
* fix import of non-existent function
* register routes...
* [Files] PR feedback 1 (#138417)
* update comments and plugin description
* filter lists and finds on non-deleted files
* updated comment
* added tests for larger files and files exactly divisible by chunk size
* [CI] Auto-commit changed files from 'node scripts/build_plugin_list_docs'
* do not use regexp
* fix test assertion
* update content stream tests
* make fewer fields searchable
* ok
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* remove legacy comment
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Vadim Kibana <82822460+vadimkibana@users.noreply.github.com>
* docs(NA): drop support for windows native development
* docs(NA): improve previous written instructions around windows de
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [kbn/pm] rewrite to avoid needing a build process
* uncomment timing reporting
* throw in a few missing comments
* Update README.md
* remove extra SomeDevLog interface from ci-stats-core
* remove non-stdio logging from bazel_runner, improve output formatting
* use private fields instead of just ts private props
* promote args to a positional arg
* optionally require the ci-stats-reporter after each command
* allow opt-ing out of vscode config management
* reduce to a single import
* add bit of docs regarding weird imports and package deps of kbn/pm
* clean extraDirs from Kibana's package.json file too
* tweak logging of run-in-packages to use --quiet and not just CI=true
* remove unlazy-loader
* add readme for @kbn/yarn-lock-validator
* convert @kbn/some-dev-logs docs to mdx
* remove missing navigation id and fix id in dev-cli-runner docs
* fix title of some-dev-logs docs page
* typo
* Add kubernetes link on nav sidebar
* Create new plugin for k8s vis page
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Paulo Henrique <paulo.scape@gmail.com>
Sets up the boilerplate code for the aiops plugin and adds a demo page within the ML app to demonstrate single API request data streaming from Kibana server to UI client.
* [ftr] automatically determine config run order
* split lens config into two groups
* support ftr configs always running against CI
* Split detection_engine_api_integration rule exception list tests
* Add configs from previous commit
* [ftr] remove testMetadata and maintain a unique lifecycle instance per run
* Revert "[ftr] remove testMetadata and maintain a unique lifecycle instance per run"
This reverts commit d2b4fdb824.
* Split alerting_api_integration/security_and_spaces tests
* Add groups to yaml
* Revert "Revert "[ftr] remove testMetadata and maintain a unique lifecycle instance per run""
This reverts commit 56232eea68.
* stop ES more forcefully and fix timeout
* only cleanup lifecycle phases when the cleanup is totally complete
* only use kill when cleaning up an esTestInstance
* fix broken import
* fix runOptions.alwaysUseSource implementation
* fix config access
* fix x-pack/ccs config
* fix ml import file paths
* update kibana build id
* revert array.concat() change
* fix baseConfig usage
* fix pie chart data
* split up maps tests
* pull in all of group5 so that es archives are loaded correctly
* add to ftr configs.yml
* fix pie chart data without breaking legacy version
* fix more pie_chart stuff in new vis lib
* restore normal PR tasks
* bump kibana-buildkite-library
* remove ciGroup validation
* remove the script which is no longer called from checks.sh
* [CI] Auto-commit changed files from 'yarn kbn run build -i @kbn/pm'
* adapt flaky test runner scripts to handle ftrConfig paths
* fix types in alerting_api_integration
* improve flaky config parsing and use non-local var name for passing explicit configs to ftr_configs.sh
* Split xpack dashboard tests
* Add configs
* [flaky] remove key from ftr-config steps
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
* restore cypress builds
* remove ciGroups from FTR config files
* fixup some docs
* add temporary script to hunt for FTR config files
* use config.base.js naming for clarity
* use script to power ftr_configs.yml
* remove usage of removed x-pack/scripts/functional_tests
* fix test names in dashboard snapshots
* bump kibana-buildkite-library
* Try retrying only failed configs
* be a little quieter about trying to get testStats from configs with testRunners defined
* Remove test code
* bump kibana-buildkite-library
* update es_snapshot and on_merge jobs too
* track duration and exit code for each config and print it at the end of the script
* store results in order, rather than by key, in case there are duplicates in $config
* bash is hard
* fix env source and use +e rather than disabling e for whole file
* bash sucks
* print config summary in jest jobs too
* define results in jest_parallel.sh
* simplify config summary print, format times a little better
* fix reference to unbound time variable, use better variable name
* skip the newline between each result
* finish with the nitpicking
* sync changes with ftr_configs.sh
* refuse to execute config files which aren't listed in the .buildkite/ftr_configs.yml
* fix config.edge.js base config import paths
* fix some readmes
* resolve paths from ftr_configs manifest
* fix readConfigFile tests
* just allow __fixtures__ configs
* list a few more cypress config files
* install the main branch of kibana-buildkite-library
* split up lens group1
* move ml data_visualizer tests to their own config
* fix import paths
* fix more imports
* install specific commit of buildkite-pipeline-library
* sort configs in ftr_configs.yml
* bump kibana-buildkite-library
* remove temporary script
* fix env var for limiting config types
* Update docs/developer/contributing/development-functional-tests.asciidoc
Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
* produce a JUnit report for saved objects field count
* apply standard concurrency limits from flaky test runner
* support customizing FTR concurrency via the env
Co-authored-by: Brian Seeders <brian.seeders@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
* chore(NA): upgrade bazel into v5
* chore(NA): initial work to upgrade to rules v5
* chore(NA): remove @elastic/datemath jest unit test for eslint plugin imports
* chore(NA): removed packager tsc_wrapped added my mistake
* Add event annotation service structure
* adding annotation layer to lens. passing event annotation service
* simplify initial Dimensions
* add annotations to lens
* no datasource layer
* group the annotations into numerical icons
* color icons in tooltip, add the annotation icon, fix date interval bug
* display old time axis for annotations
* error in annotation dimension when date histogram is removed
* refactor: use the same methods for annotations and reference lines
* wip
* only check activeData for dataLayers
* added new icons for annotations
* refactor icons
* uniqueLabels
* unique Labels
* diff config from args
* change timestamp format
* added expression event_annotation_group
* names refactor
* ea service adding help descriptions
* rotate icon
* added tests
* fix button problem
* dnd problem
* dnd fix
* tests for dimension trigger
* tests for unique labels
* [CI] Auto-commit changed files from 'node scripts/build_plugin_list_docs'
* type
* add new button test
* remove noDatasource from config (only needed when initializing a layer or dimension in getSupportedLayers)
* addressing Joe's and Michael comments
* remove hexagon and square, address Stratoula's feedback
* stroke for icons & icon fill
* fix tests
* fix small things
* align the set with tsvb
* align IconSelect
* fix i18nrc
* Update src/plugins/event_annotation/public/event_annotation_service/index.tsx
Co-authored-by: Alexey Antonov <alexwizp@gmail.com>
* refactor empty button
* CR
* date cr
* remove DimensionEditorSection
* change to emptyShade for traingle fill
* Update x-pack/plugins/lens/public/app_plugin/app.scss
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Alexey Antonov <alexwizp@gmail.com>
With a recent virtualbox update, we're getting failures in CI with:
> The IP address configured for the host-only network is not within the
> allowed ranges. Please update the address used to be within the allowed
> ranges and run the command again.
This updates the IP range used to be compatible.
* Add new plugint to collect additional kibana monitoring metrics
* Readme
* Update generated document
* We won't use this route
* Use dynamic route style
* Add in mapping verification
* Fix types
* Feedback from PR
* PR feedback
* We do not need this
* PR feedback
* Match options to api/stats
* PR feedback
* Ensure we always require auth
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
