* WIP: Adding new route for EQL Validation
This is mostly boilerplate with some rough parameter definitions; the
actual implementation of the validation is going to live in our
validateEql function.
A few tests are failing as the mocks haven't yet been implemented, I
need to see the shape of the responses first.
* Cherry-pick Marshall's EQL types
* Implements actual EQL validation
* Performs an EQL search
* filters out non-parsing errors, and returns what remains in the
response
* Adds mocks for empty EQL responses (we don't yet have a need for
mocked data, but we will when we unit-test validateEql)
* Adds validation calls to the EQL form input
* Adds EQL Validation response schema,mocks,tests
* Adds frontend function to call our validation endpoint
* Adds hook, useEqlValidation, to call the above function and return
state
* Adds labels/help text for EQL Query bar
* EqlQueryBar consumes useEqlValidation and marks the field as invalid,
but does not yet report errors.
* Do not call the validation API if query is not present
This causes a broader error that results in a 400 response; we can (and
do) handle the case of a blank query in the form itself.
* Remove EQL Help Text
It doesn't add any information for the user, and it currently looks bad
when combined with validation errors.
* Flesh out and use our popover for displaying validation errors
* Fixes issue where old errors were persisted after the user had made
modifications
* Include verification_exception errors as validation errors
These include errors related to index fields and mappings.
* Generalize our validation helpers
We're concerned with validation errors; the source of those errors is an
implementation detail of these functions.
* Move error popover and EQL reference link to footer
This more closely resembles the new Eui Markdown editor, which places
errors and doc links in a footer.
* Fix jest tests following additional prop
* Add icon for EQL Rule card
* Fixes existing EqlQueryBar tests
These were broken by our use of useAppToasts and the EUI theme.
* Add unit tests around error rendering on EQL Query Bar
* Add tests for ErrorPopover
* Remove unused schema type
Decode doesn't do any additional processing, so we can use t.TypeOf here
(the default for buildRouteValidation).
* Remove duplicated header
* Use ignore parameter to prevent EQL validations from logging errors
Without `ignore: [400]` the ES client will log errors and then throw
them. We can catch the error, but the log is undesirable.
This updates the query to use the ignore parameter, along with updating
the validation logic to work with the updated response.
Adds some mocks and tests around these responses and helpers, since
these will exist independent of the validation implementation.
* Include mapping_exceptions during EQL query validation
These include errors for inaccessible indexes, which should be useful to
the rule writer in writing their EQL query.
* Display toast messages for non-validation messages
* fix type errors
This type was renamed.
* Do not request data in our validation request
By not having the cluster retrieve/send any data, this should saves us
a few CPU cycles.
* Move EQL validation to an async form validator
Rather than invoking a custom validation hook (useEqlValidation) at custom times (onBlur) in our EqlQueryBar
component, we can instead move this functionality to a form validation
function and have it be invoked automatically by our form when values
change. However, because we still need to handle the validation messages
slightly differently (place them in a popover as opposed to an
EuiFormRow), we also need custom error retrieval in the form of
getValidationResults.
After much pain, it was determined that the default behavior of
_.debounce does not work with async validator functions, as a debounced
call will not "wait" for the eventual invocation but will instead return
the most recently resolved value. This leads to stale validation
results and terrible UX, so I wrote a custom function (debounceAsync)
that behaves like we want/need; see tests for details.
* Invalidate our query field when index patterns change
Since EQL rules actually validate against the relevant indexes, changing
said indexes should invalidate/revalidate the query.
With the form lib, this is beautifully simple :)
* Set a min-height on our EQL textarea
* Remove unused prop from EqlQueryBar
Index corresponds to the value from the index field; now that our EQL
validation is performed by the form we have no need for it here.
* Update EQL overview link to point to elasticsearch docs
Adds an entry in our doclinks service, and uses that.
* Remove unused prop from stale tests
* Update docLinks documentation with new EQL link
* Fix bug where saved query rules had no type selected on Edit
* Wait for kibana requests to complete before moving between rule tabs
With our new async validation, a user can quickly navigate away from the
Definition tab before the validation has completed, resulting in the
form being invalidated. Any subsequent user actions cause the form to
correct itself, but until I can find a better solution here this really
just gives the validation time to complete and sidesteps the issue.
Removes the `npStart` legacy singleton used during the migration to the new platform. The singleton was used in API calls to access the `http.fetch` service. To remove the singleton we have injected `fetch` as a dependency in all functions.
* [APM] Fix anomalies not showing up on transaction charts
* Added API tests to check transaction groups charts for anomaly data
* Improve test names and assertions from PR feedback
* Updated the transaction groups chart API to make `environment` a
required param while making `uiFilters` optional
* updates the basic API tests for transaction_groups/charts with the
required `environment` param
* makes uiFIltersES default to [] on core setup and removes SetupUIFilters type
* fixes vertical shade
* - replaces uiFiltersES with esFilter & uiFilters and cleans up related code around these
- deduplicates the required environment in the transaction_groups/charts API
* updates basic apm_api_integration tests
* pr feedback
* updates api test snapshots with correct anomaly data
* removed environment query param from useTransactionCharts and ensures
it's included in uiFilters returned from useUrlParams
Co-authored-by: Oliver Gupte <olivergupte@gmail.com>
* chore(NA): script to check for multiple version of same dependency
* chore(NA): remove multiple versions for the same dependency
* chore(NA): hook single_version_dependencies script into the CI
* chore(NA): remove grunt from the CI hook integration
* chore(NA): update kbn pm dist
* chore(NA): fix typechecking
* chore(NA): update code to run under last extract-zip version
* fix(NA): multiple versions of the same type dependency
* move validation to bootstrap (#13)
Co-authored-by: spalger <spalger@users.noreply.github.com>
* chore(NA): todo to remove logic to validate single version deps once we move into a single package.json
* chore(NA): remove verify dependency versions jenkins task
* chore(NA): update kbn pm dist file
* chore(NA): remove last mention to verify_dependency_versions.sh fom tasks.groovy
Co-authored-by: Spencer <email@spalger.com>
Co-authored-by: spalger <spalger@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
This adds threat matching rule type to the rule creator.
Screen shot of creating a threat match
<img width="1023" alt="Screen Shot 2020-09-30 at 3 31 09 PM" src="https://user-images.githubusercontent.com/1151048/94742158-791b1c00-0332-11eb-9d79-78ab431322f0.png">
---
Screen shot of the description after creating one
<img width="1128" alt="Screen Shot 2020-09-30 at 3 29 32 PM" src="https://user-images.githubusercontent.com/1151048/94742203-8b955580-0332-11eb-837f-5b4383044a13.png">
---
Screen shot of first creating a threat match without values filled out
<img width="1017" alt="Screen Shot 2020-09-30 at 3 27 29 PM" src="https://user-images.githubusercontent.com/1151048/94742222-95b75400-0332-11eb-9872-e7670e917941.png">
Additions and bug fixes:
* Changes the threat index to be an array
* Adds a threat_language to the REST schema so that we can use KQL, Lucene, (others in the future)
* Adds plumbing for threat_list to work with the other REST endpoints such as PUT, PATCH, etc...
* Adds the AND, OR dialog and user interface
**Usage**
If you are a team member using the team servers you can skip this usage section of creating threat index. Otherwise if you want to know how to create a mock threat index, instructions are below.
Go to the folder:
```ts
/kibana/x-pack/plugins/security_solution/server/lib/detection_engine/scripts
```
And post a small ECS threat mapping to the index called `mock-threat-list`:
```ts
./create_threat_mapping.sh
```
Then to post a small number of threats that represent simple port numbers you can run:
```ts
./create_threat_data.sh
```
However, feel free to also manually create them directly in your dev tools like so:
```ts
# Posts a threat list item called some-name with an IP but change these out for valid data in your system
PUT mock-threat-list-1/_doc/9999
{
"@timestamp": "2020-09-09T20:30:45.725Z",
"host": {
"name": "some-name",
"ip": "127.0.0.1"
}
}
```
```ts
# Posts a destination port number to watch
PUT mock-threat-list-1/_doc/10000
{
"@timestamp": "2020-09-08T20:30:45.725Z",
"destination": {
"port": "443"
}
}
```
```ts
# Posts a source port number to watch
PUT mock-threat-list-1/_doc/10001
{
"@timestamp": "2020-09-08T20:30:45.725Z",
"source": {
"port": "443"
}
}
```
### Checklist
- [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
- [ ] This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)
- [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)
- [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
resolves https://github.com/elastic/kibana/issues/51099
This formalizes the concept of "alert status", in terms of it's execution, with
some new fields in the alert saved object and types used with the alert client
and http APIs.
These fields are read-only from the client point-of-view; they are provided in
the alert structures, but are only updated by the alerting framework itself.
The values will be updated after each run of the alert type executor.
The data is added to the alert as the `executionStatus` field, with the
following shape:
```ts
interface AlertExecutionStatus {
status: 'ok' | 'active' | 'error' | 'pending' | 'unknown';
lastExecutionDate: Date;
error?: {
reason: 'read' | 'decrypt' | 'execute' | 'unknown';
message: string;
};
}
```
* Extended Email action configuration with hasAuth property to identify if the connector require user credentials. Improved UX for Email connector
* Fixed failing tests and comments
* Fixed type check and reverted logic of Add user and password switch button
* Fixed due to the latest design requirenments
* Fixed due to review comments
- Fixes cell color coding based on influence score for outlier detection results page data grid. (Part of #77046)
- Introduces expandable sections (<ExpandableSection />). In contrast to plain accordions, the main idea of this component is that it should also provide some sort of useful summary when collapsed instead of just being an expandable title. For example, the "Analysis" section is collapsed by default, but still offers information like analysis type, source and destination index. This concept should allow us to keep the analytics results pages usable with more content (additional results, evaluations, visualizations) being added over time.
- The "Analysis" section is a reuse of the expandable row from the analytics jobs list. Some design adjustments have been made to make it usable in both places.
