Commit graph

460 commits

Author SHA1 Message Date
Melissa Alvarez
bd13284bfe
[ML] Ensure job group badge fonts are same color (#86674)
* use badge component to keep group badge consistent. sort groups

* use all colors now that badge component is used

* set left margin on group badge wrapped in link to match badge margin

* add badge colors to edit flyout job groups
2020-12-22 12:41:18 -05:00
James Gowdy
6d72042ca4
[ML] Fixing endpoint schema for can_delete_job endpoint (#86436)
* [ML] Fixing endpoint schema for can_delete_job endpoint

* changing get to post in docs

* renaming canUntag

* fixing typo

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2020-12-21 12:17:53 +00:00
Thomas Watson
e8b21bc6c1
Upgrade to hapi version 20 (#85406) 2020-12-19 13:10:11 +01:00
James Gowdy
b1d18e5d9e
[ML] Fixing job id reuse when previous job was in diffrent space (#86270) 2020-12-18 09:40:27 +00:00
Walter Rafelsberger
755dc7e8ee
[ML] Anomaly Detection: Fix validation error when no data in index. (#86114)
For cardinality checks, an empty index of the fields checks not returning any results would block the user from moving to the next step in the Anomaly Detection job wizard.
This PR fixes it by adding more fine grained checks and only returning warning-level messages for the above cases. A warning-level message will allow the user to continue to the next step in the wizard.
2020-12-17 18:58:37 +01:00
James Gowdy
6f8697e198
[ML] Removing space awareness for trained models (#86132) 2020-12-17 14:21:31 +00:00
Pete Harverson
b70cabbdcb
[ML] Fix sort order of data recognizer module cards (#86250)
* [ML] Fix sort order of data recognizer module cards

* [ML] Add sort by ID to server side recognize endpoint

* [ML] Enhance docs for recognize endpoint success response.
2020-12-17 14:11:51 +00:00
James Gowdy
2066b3d7ca
[ML] Fix synching jobs from other spaces (#86086)
* [ML] Fix synching jobs from other spaces

* updating text in delete modal

* updating translations

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2020-12-17 09:08:17 +00:00
James Gowdy
ed250faabc
[ML] Adding runtime_mappings to job wizards (#85817)
* [ML] Adding runtime_mappings to job wizards

* fixing test

* adding runtime fields to fields list in datafeed preview

* fixing cardinality count for runtime field

* fixing cardinality check in data visualizer

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2020-12-16 14:55:01 +00:00
Melissa Alvarez
43e20112b1
[ML] Data Frame Analytics: check space permissions before deleting jobs (#85495)
* disable delete until checks complete

* add canDeleteJobs wrapper in saved objects service

* create DeleteJobCheckModal shared component

* wip: add deleteJobModal check in list and map views

* adding remove from current space endpoint

* updating error text

* fixing typo in variable name

* Update button content. Add untagging functionality

* adding anomaly detection delete job modal

* fix modal content bug

* refresh job map after deletion or untagging

* adding job refresh to anomaly detectors

* go straight to delete flow if only action available

* fixing test

* update text

* increase line spacing in check modal

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: James Gowdy <jgowdy@elastic.co>
2020-12-15 12:44:03 -05:00
Pete Harverson
bf869422e1
[ML] Only show _doc_count for anomaly detection job summary count field (#85444)
* [ML] Only show _doc_count for anomaly detection job summary count field

* [ML] Use array filter for removing _DOC_COUNT field
2020-12-10 22:30:39 +00:00
Craig Chamberlain
313d85e985
[ML] Adds security_linux and security_windows Modules (#85065)
* initial commit

refactored multi-index, multi-pipeline jobs for 7.11. These are new modules that will live alongside the existing jobs.

* Update ml_modules.tsx

added new module names to the list

* Update get_module.ts

added new module names

* Linter fixes

* Order matters

* manifest fixes

added colon char to the module name and shortened the description

* additon to description

after talking with the security team today, adding this suggested text to the beginning of the description so it will tend to be visible to the user:
"This is a new refactored job which works on ECS compatible events across multiple indices."

* Adjust module recognizer test for auditbeat dataset

* influencers

changes to the metadata jobs to make influencers identical to the originals

* change for security app

changes to two datafeeds needed for logic in the Security app - added the suffix "_ecs" to two ids.

Co-authored-by: Garrett Spong <spong@users.noreply.github.com>
Co-authored-by: Robert Oskamp <robert.oskamp@elastic.co>
2020-12-10 14:02:41 -05:00
Quynh Nguyen
008a420f81
[ML] Improve messaging and support for datafeed using aggregated and scripted fields (#84594) 2020-12-10 11:35:51 -06:00
Melissa Alvarez
93670ec81f
[ML] Data frame analytics: Adds job deletion and creation to map view (#84299)
* wip: delete job node and update map

* adds ability to delete job and update map

* create job from index node

* can clone job from map

* reset map button

* remove trained model node when deleting job

* remove related model node. remove map tab when root node deleted

* ensure model with no job shows up correctly

* update types and naming

* use urlGenerator

* fix inner scrollbar

* Adjust cytoscapeOptions after EUI update

Co-authored-by: Robert Oskamp <robert.oskamp@elastic.co>
2020-12-09 10:11:08 -05:00
James Gowdy
090359ca5a
[ML] Renaming saved object repair to sync (#84311)
* [ML] Remaiming saved object repair to sync

* removing 0 from sync lists

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2020-11-30 12:24:03 +00:00
James Gowdy
767286e2da
[ML] Fix spaces job ID check (#84404)
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2020-11-30 09:34:33 +00:00
James Gowdy
24f262b9ca
[ML] Space permision checks for job deletion (#83871)
* [ML] Space permision checks for job deletion

* updating spaces dependency

* updating endpoint comments

* adding delete job capabilities check

* small change based on review

* improving permissions checks

* renaming function and endpoint

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2020-11-24 16:29:32 +00:00
Melissa Alvarez
00e59512fa
[ML] Data frame analytics: Adds functionality to map view (#83710)
* get all jobs from index node

* create map from modelId and enable url share

* highlight source node

* add map endpoint to api doc

* use variables in css.fix types.ensure map tab is shown

* fix translations
2020-11-20 10:39:30 -05:00
Larry Gregory
7f962e5839
Removing circular dependency between spaces and security (#81891)
* Removing circular dependency between spaces and security

* Apply suggestions from code review

Co-authored-by: Constance <constancecchen@users.noreply.github.com>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>

* Tests refactor

- Reorganize top level describes into 3 space-based blocks into based on spaces:
  - space disabled
  - spaces plugin unavailable
  - space enabled (most previous tests go under this new block) with new beforeEach

- wrote new tests for uncovered lines 58, 66-69

* Review1: address PR feedback

* changing fake requests for alerts/actions

* Fixing tests

* fixing more tests

* Additional testing and refactoring

* Apply suggestions from code review

Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>

* Review 2: Address feedback

* Make ESLint happy again

Co-authored-by: Constance <constancecchen@users.noreply.github.com>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>
Co-authored-by: Constance Chen <constance.chen.3@gmail.com>
2020-11-19 13:41:13 -05:00
James Gowdy
78d7bfdf97
[ML] Space management UI (#83320)
* [ML] Space management UI

* fixing types

* small react refactor

* adding repair toasts

* text and style changes

* handling spaces being disabled

* correcting initalizing endpoint response

* text updates

* text updates

* fixing spaces manager use when spaces is disabled

* more text updates

* switching to delete saved object first rather than overwrite

* filtering non ml spaces

* renaming file

* fixing types

* updating list style

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2020-11-19 15:26:01 +00:00
Quynh Nguyen
55119c2152
[ML] Improve support for script and aggregation fields in anomaly detection jobs (#81923)
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2020-11-17 11:34:15 -06:00
James Gowdy
6a1cd73082
[ML] Additional job spaces initialization (#83127)
* [ML] Additional job spaces initialization

* adding logs test

* updating integrations

* updating test text

* fixing logs jobs error

* fix bug with duplicate ids

* updating initialization log text

* fixing initialization text

* adding metrics overrides

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2020-11-16 21:56:41 +00:00
Nathan L Smith
3f51bf5e9f
Fix advanced settings category sorting (#83394)
In the advanced settings categories, "Observability" and "Machine Learning" were using uppercase letters in their keys while everything else was using lowercase

This caused them both to show up before the rest of the options in the dropdown and in the sorting in the advanced settings UI.

Add keys for them to the get_category_name module in the advanced settings plugin and use those keys in the plugins that apply these categories.

This also makes it so i18n keys are available for these items.

Fixes #81974.
2020-11-16 09:23:28 -06:00
Melissa Alvarez
6519b83e48
[ML] Data frame analytics: Adds map view (#81666)
* add analytics map endpoint and server model

* add map action to job and models list

* wip:fetch models for jobs. Use url generator

* get models when extending node. deduplicate elements

* add job type icons. disable map action if job not finished.

* move shared const to common dir

* persist map tab. handle indexPattern from visualizer

* use url generator in models list

* temporarily disable delete action in flyout

* update legend style. make map horizontal

* update dfa model to use spaces changes

* format creation time

* update from indexPattern to index.remove refresh button

* handle index patterns with wildcard
2020-11-12 12:40:08 -05:00
Dima Arnautov
6e9177caac
[ML] Update apidoc config with the Trained models endpoints (#83274)
* [ML] fix apidoc annotations

* [ML] add trained models

* [ML] use full path to the apidoc-markdown package
2020-11-12 17:51:42 +01:00
Dima Arnautov
c32215d7ff
[ML] Fix apiDocs extractor script (#82582)
* [ML] fix serializer script

* [ML] fix extractor

* [ML] bump apiDoc version

* [ML] update yarn.lock
2020-11-11 15:42:53 +01:00
James Gowdy
096acb4da8
[ML] job saved objects initialization (#82639)
* [ML] job saved objects initialization

* fixing job count logic

* adding missing files

* attempting to fix build crash

* fixing kibana.json

* changes based on review

* removing accidentally added export

* adding intialization promise

* use finally so errors dont stop initialization

* function rename

* removing duplicate header

* adding job initialization count to log message

* adding error to log message

* moving initialization file

* moving intialization file back again to fix git stash issues

* removing .kibana index search

* creating internal saved object client

* code clean up

* removing commented code

* adding check for spaces enabled

* adding ids to saved objects

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2020-11-10 21:02:09 +00:00
James Gowdy
dc287eaadc
[ML] Adding non-space aware checks for existing jobs (#82814)
* [ML] Adding non-space aware checks for existing jobs

* merge with existing jobs exist endpoint

* updating comments

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2020-11-10 20:54:32 +00:00
Quynh Nguyen
9c984f4723
[ML] Add option for anomaly charts for metric detector should plot min, mean or max as appropriate (#81662) 2020-11-09 10:41:04 -06:00
Melissa Alvarez
ae20a3a2a9
[ML] Add unsigned_long support to data frame analytics and anomaly detection (#82636)
* add support for unsigned_long field in dfa

* add support for unsigned_long for anomaly detection
2020-11-06 16:51:14 -05:00
James Gowdy
f4386fc5b0
[ML] Updating analysis config schema (#82703)
* [ML] Updating analyss config schema

* better schema order
2020-11-05 13:41:16 +00:00
Quynh Nguyen
b8307b498c
[ML] Add probability values in decision path visualization for classification data frame analytics (#80229)
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2020-11-03 18:47:21 -06:00
James Gowdy
a0fded52eb
[ML] Adding space aware jobs (#77916)
* [ML] Adding space aware jobs

* adding mlClient

* switching to type includes

* adding additional job checks

* fixing conflict

* adding dfa checks

* refactoring jobs in spaces checks

* filtering calendars

* adding initial job object status and repair endpoints

* enabling repair endpoint

* fixing listed jobs in status

* adding datafeed repair

* updating shared services

* adding results job id check

* fixing conflicts

* don't remove SO on delete

* fixing non-ml plugins

* filtering job audit messages

* fixing types

* fixing tests

* adding job ids wildcard support

* removing empty migration test

* fixing tests and disabling spaces test user

* adding saved objects all permission

* fixing calendars

* updating job 404

* updating job wildcard search

* renaming services

* fixing conflicts

* fixing log tests

* disabling apm test

* skipping more apm tests

* optimzing repair

* fixing types

* updating apm test archive to include ML saved objects

* enabling disabled test

* removing comment

* adding space assigning endpoints

* adding saved object default permissions

* removing commented code

* loading all jobs for all spaces for status check

* adding spaces list endpoint

* adding job spaces to management page

* adding trained model fltering

* fixing trained model id check and job wildcard check

* fixing types

* fixing bug when adding new job to calendar

* changes based on review

* updating schema

* changes based on review

* fixing types

* rolling back http service injection

* fixing http service injection

* adding errrors to repair endpoint response

* updating api doc

* improving types

* disabling id check on ad get endpoints

* fixing tests

* fixing group requests

* adding comments

* using filter in saved object search

* fixing fake request issue

* removing console log

* making job saved object hidden

* removing acccidentally included file

* renaming saved object client

* updating apidoc

* unhiding ml saved objects

* moving route guard

* improving error when SOC is null

* fixing types after merge with master

* fixing tests

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2020-11-03 15:34:35 +00:00
Dima Arnautov
f50c5a2cd3
[ML] Configure sorting for partition values on Single Metric Viewer (#81510)
* [ML] fix callout styles

* [ML] refactor timeseriesexplorer.js, add series_controls.tsx, storage support for partition config

* [ML] anomalousOnly support

* [ML] sort by control

* [ML] update query

* [ML] sort order controls

* [ML] adjust query

* [ML] merge default and local configs, add info

* [ML] fix types, adjust sorting logic for model plot results

* [ML] fix translation keys

* [ML] fixed size for the icon flex item

* [ML] fix time range condition, refactor

* [ML] change info messages and the icon color

* Fix model plot info message

Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>

* [ML] functional tests

* [ML] rename ML_ENTITY_FIELDS_CONFIG

* [ML] support manual input

* [ML] show max record score color indicator

* [ML] use :checked selector

* [ML] refactor functional tests

* [ML] extend config with "applyTimeRange", refactor with entity_config.tsx

* [ML] info messages

* [ML] remove custom message

* [ML] adjust the endpoint

* [ML] customOptionText

* [ML] sort by name UI tweak

* [ML] change text

* [ML] remove TODO comment

* [ML] fix functional test

* [ML] move "Anomalous only"/"Apply time range" control to the bottom of the popover

* [ML] update types
2020-11-02 16:04:59 +01:00
Thomas Watson
700225061a
Upgrade to hapi version 18 (#80468) 2020-11-02 13:18:00 +01:00
Thomas Watson
1407f713e5
Update KibanaRequest to use the new WHATWG URL API (#80713) 2020-10-29 14:35:48 +01:00
Larry Gregory
bf0f8bbb42
Cleanup feature registration (#80909) 2020-10-20 12:53:43 -04:00
James Gowdy
6db108b914
[ML] Fixing exclude frequent in advanced wizard (#81121)
* [ML] Fixing exclude frequent in advanced wizard

* updating description

* adding exclude_frequent to test
2020-10-20 14:00:29 +01:00
Quynh Nguyen
f04c295d4d
[ML] Fixes anomaly detection jobs list load if call to load job messages fails (#79792)
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2020-10-08 11:12:26 -05:00
James Gowdy
57f6ecda73
[ML] Sending secondary auth headers to _explain (#79814) 2020-10-07 09:00:43 +01:00
Dima Arnautov
3002108c40
[ML] rename inference to trained_models (#79676) 2020-10-06 19:31:59 +02:00
Melissa Alvarez
06f87bb838
[ML] DF Analytics creation wizard: resolve clone usability issues (#79048)
* show error when clone fails due to no index pattern

* default results_field unless specified

* results field switch set to on if resultsField for cloned job is default

* ensure cloned job config not overwritten in advanced editor

* show errorToast if unable to clone anomalyDetection job due to no indexPattern

* ensure jobConfig query getting saved in form state

* ensure index patterns with commas handled correctly

* clone should accept comma separated index patterns

* use nullish coalescing operator when checking for undefined analysisFields
2020-10-06 13:29:16 -04:00
Quynh Nguyen
ad89e6f956
[ML] Add runtime fields support (#78700)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2020-10-05 13:00:58 -05:00
Tyler Smalley
7211f78ce1
Bumps Jest related packages (#78720)
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
2020-10-01 14:38:51 -07:00
Quynh Nguyen
31efa1ab5b
[ML] Add feature importance summary charts (#78238)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-10-01 09:39:17 -05:00
Spencer
06d1628a00
[ts] enable "resolveJsonModule" and disable existing failures (#78855)
Co-authored-by: spalger <spalger@users.noreply.github.com>
2020-09-30 15:44:43 -07:00
James Gowdy
c4f05c4758
[ML] Fixing module datafeed overrides (#78925) 2020-09-30 18:27:03 +01:00
Pete Harverson
80c004cb01
[ML] Replace use of rest_total_hits_as_int with track_total_hits (#78423)
* [ML] Replace use of rest_total_hits_as_int with track_total_hits

* [ML] Replace use of lodash get for obtaining hits total

* [ML] Update mocks to use new total hits format
2020-09-30 08:54:17 +01:00
Walter Rafelsberger
61aac716b3
[ML] Limit exposing shared static code through ml/public/index.ts. (#77745)
- Clean up lodash imports.
- Import types only where applicable.
- Reduce page load bundle size by fetching more code asynchronously.
2020-09-29 11:57:53 +02:00
Felix Stürmer
3f2e9f7705
[Logs UI] Add dataset-specific categorization warnings (#75351)
This adds dataset-specific categorization warnings for the categorization module. The warnings are displayed in call-outs on the relevant tabs as well as the job setup screens if a prior job with warnings exists. To that end this also changes the categorization job configuration to enable the partitioned categorization mode.

Co-authored-by: Alejandro Fernández Gómez <antarticonorte@gmail.com>
2020-09-24 17:02:23 +02:00