github-mirrors/kibana
1
0
Fork 0
mirror of https://github.com/elastic/kibana.git synced 2025-04-24 17:59:23 -04:00
Code Issues Projects Releases Packages Wiki Activity

Update Kuery Syntax (#15857)

Browse source 
This PR evolves Kuery into a simplified, more lucene-like language. In a follow up PR we plan on getting rid of the language switcher dropdown and instead add a checkbox that will allow users to opt-in to this syntax, which will feel less like a separate language and more like a set of enhancements to the existing lucene query syntax.

This PR also updates the simple grammar to solve the following:

* Wildcards in field names
* "is one of" shorthand (e.g. foo:(bar OR baz OR qux))
* Don't split on whitespace (require explicit AND/ORs, case insensitive)
* "exists" query (e.g. foo:*)
* Improved range query shorthands (e.g. foo <= 1)
* Wildcard queries

Since this new syntax is simpler and does not support every filter type like old Kuery did, this PR also brings back the filter bar when kuery is selected in the language picker.

See the documentation updates in this PR if you need an overview of the new syntax (and let us know if the documentation is lacking).
This commit is contained in:
Lukas Olson 2018-03-07 08:29:34 -08:00 • committed by Matt Bargar
parent aeadf67d74
commit e9ff31f185
44 changed files with 982 additions and 1076 deletions
Download patch file Download diff file Expand all files Collapse all files

117
docs/discover/kuery.asciidoc
View file

@ -3,22 +3,32 @@
experimental[This functionality is experimental and may be changed or removed completely in a future release.]
[NOTE]
============
Breaking changes were made to Kuery's experimental syntax in 6.3. Read on for details of the new syntax.
============
Kuery is a new query language built specifically for Kibana. It aims to simplify the search experience in Kibana
and enable the creation of helpful features like auto-complete, seamless migration of saved searches, additional
query types, and more. Kuery is a basic experience today but we're hard at work building these additional features on
top of the foundation Kuery provides.
Kueries are built with functions. Many functions take a field name as their first argument. Extremely common functions have shorthand notations.
If you're familiar with Kibana's old lucene query syntax, you should feel right at home with Kuery. Both languages
are very similar, but there are some differences we'll note along the way.
`is("response", 200)` will match documents where the response field matches the value 200.
`response:200` does the same thing. `:` is an alias for the `is` function.
`response:200` will match documents where the response field matches the value 200.
Multiple search terms are separated by whitespace.
Quotes around a search term will initiate a phrase search. For example, `message:"Quick brown fox"` will search
for the phrase "quick brown fox" in the message field. Without the quotes, your query will get broken down into tokens via
the message field's configured analyzer and will match documents that contain those tokens, regardless of the order in which
they appear. This means documents with "quick brown fox" will match, but so will "quick fox brown". Remember to use quotes if you want
to search for a phrase.
`response:200 extension:php` will match documents where response matches 200 and extension matches php.
Unlike lucene, Kuery will not split on whitespace. Multiple search terms must be separated by explicit
boolean operators. Note that boolean operators in Kuery are not case sensitive.
*All terms must match by default*. The language supports boolean logic with and/or operators. The above query is equivalent to `response:200 and extension:php`.
This is a departure from the Lucene query syntax where all terms are optional by default.
`response:200 extension:php` in lucene would become `response:200 and extension:php`.
This will match documents where response matches 200 and extension matches php.
We can make terms optional by using `or`.
@ -32,85 +42,40 @@ We can override the default precedence with grouping.
`response:200 and (extension:php or extension:css)` will match documents where response is 200 and extension is either php or css.
Terms can be inverted by prefixing them with `!`.
A shorthand exists that allows us to easily search a single field for multiple values.
`!response:200` will match all documents where response is not 200.
`response:(200 or 404)` searches for docs where the `response` field matches 200 or 404. We can also search for docs
with multi-value fields that contain a list of terms, for example: `tags:(success and info and security)`
Terms can be inverted by prefixing them with `not`.
`not response:200` will match all documents where response is not 200.
Entire groups can also be inverted.
`response:200 and !(extension:php or extension:css)`
`response:200 and not (extension:php or extension:css)`
Some query functions have named arguments.
Ranges in Kuery are similar to lucene with a small syntactical difference.
`range("bytes", gt=1000, lt=8000)` will match documents where the bytes field is greater than 1000 and less than 8000.
Instead of `bytes:>1000`, Kuery omits the colon: `bytes > 1000`.
`>, >=, <, <=` are all valid range operators.
Exist queries are simple and do not require a special operator. `response:*` will find all docs where the response
field exists.
Wildcard queries are available. `machine.os:win*` would match docs where the machine.os field starts with "win", which
would match values like "windows 7" and "windows 10".
Wildcards also allow us to search multiple fields at once. This can come in handy when you have both `text` and `keyword`
versions of a field. Let's say we have `machine.os` and `machine.os.keyword` fields and we want to check both for the term
"windows 10". We can do it like this: `machine.os*:windows 10".
Quotes are generally optional if your terms don't have whitespace or special characters. `range(bytes, gt=1000, lt=8000)`
would also be a valid query.
[NOTE]
============
Terms without fields will be matched against all fields. For example, a query for `response:200` will search for the value 200
Terms without fields will be matched against the default field in your index settings. If a default field is not
set these terms will be matched against all fields. For example, a query for `response:200` will search for the value 200
in the response field, but a query for just `200` will search for 200 across all fields in your index.
============
==== Function Reference
[horizontal]
Function Name:: Description
and::
Purpose::: Match all given sub-queries
Alias::: `and` as a binary operator
Examples:::
* `and(response:200, extension:php)`
* `response:200 and extension:php`
or::
Purpose::: Match one or more sub-queries
Alias::: `or` as a binary operator
Examples:::
* `or(extension:css, extension:php)`
* `extension:css or extension:php`
not::
Purpose::: Negates a sub-query
Alias::: `!` as a prefix operator
Examples:::
* `not(response:200)`
* `!response:200`
is::
Purpose::: Matches a field with a given term
Alias::: `:`
Examples:::
* `is("response", 200)`
* `response:200`
range::
Purpose::: Match a field against a range of values.
Alias::: `:[]`
Examples:::
* `range("bytes", gt=1000, lt=8000)`
* `bytes:[1000 to 8000]`
Named arguments:::
* `gt` - greater than
* `gte` - greater than or equal to
* `lt` - less than
* `lte` - less than or equal to
exists::
Purpose::: Match documents where a given field exists
Examples::: `exists("response")`
geoBoundingBox::
Purpose::: Creates a geo_bounding_box query
Examples:::
* `geoBoundingBox("coordinates", topLeft="40.73, -74.1", bottomRight="40.01, -71.12")` (whitespace between lat and lon is ignored)
Named arguments:::
* `topLeft` - the top left corner of the bounding box as a "lat, lon" string
* `bottomRight` - the bottom right corner of the bounding box as a "lat, lon" string
geoPolygon::
Purpose::: Creates a geo_polygon query given 3 or more points as "lat, lon"
Examples:::
* `geoPolygon("geo.coordinates", "40.97, -127.26", "24.20, -84.375", "40.44, -66.09")`

1
src/core_plugins/kibana/public/dashboard/dashboard_app.html
View file

@ -39,7 +39,6 @@
ng-show="showFilterBar()"
state="state"
index-patterns="indexPatterns"
ng-if="['lucene', 'kql'].includes(model.query.language)"
></filter-bar>
<div

1
src/core_plugins/kibana/public/discover/index.html
View file

@ -39,7 +39,6 @@
<filter-bar
state="state"
index-patterns="[indexPattern]"
ng-if="['lucene', 'kql'].includes(state.query.language)"
></filter-bar>
</div>
<div class="row">

2
src/core_plugins/kibana/public/visualize/editor/editor.html
View file

@ -50,7 +50,7 @@
<!-- Filters. -->
<filter-bar
ng-if="vis.type.options.showFilterBar && ['lucene', 'kql'].includes(state.query.language)"
ng-if="vis.type.options.showFilterBar"
state="state"
index-patterns="[indexPattern]"
></filter-bar>

36
src/core_plugins/tile_map/public/coordinate_maps_visualization.js
View file

@ -117,39 +117,11 @@ export function CoordinateMapsVisualizationProvider(Notifier, Private) {
const indexPatternName = agg.vis.indexPattern.id;
const field = agg.fieldName();
const query = this.vis.API.queryManager.getQuery();
const language = query.language;
const filter = { meta: { negate: false, index: indexPatternName } };
filter[filterName] = { ignore_unmapped: true };
filter[filterName][field] = filterData;
if (['lucene', 'kql'].includes(language)) {
const filter = { meta: { negate: false, index: indexPatternName } };
filter[filterName] = { ignore_unmapped: true };
filter[filterName][field] = filterData;
this.vis.API.queryFilter.addFilters([filter]);
}
else if (language === 'kuery') {
const { fromKueryExpression, toKueryExpression, nodeTypes } = this.vis.API.kuery;
let newQuery;
if (filterName === 'geo_bounding_box') {
newQuery = nodeTypes.function.buildNode('geoBoundingBox', field, _.mapKeys(filterData, (value, key) => _.camelCase(key)));
}
else if (filterName === 'geo_polygon') {
newQuery = nodeTypes.function.buildNode('geoPolygon', field, filterData.points);
}
else {
throw new Error(`Kuery does not support ${filterName} queries`);
}
const allQueries = _.isEmpty(query.query)
? [newQuery]
: [fromKueryExpression(query.query), newQuery];
this.vis.API.queryManager.setQuery({
query: toKueryExpression(nodeTypes.function.buildNode('and', allQueries, 'implicit')),
language: 'kuery'
});
}
this.vis.API.queryFilter.addFilters([filter]);
this.vis.updateState();
}

1
src/test_utils/public/stub_index_pattern.js
View file

@ -16,6 +16,7 @@ export default function (Private) {
function StubIndexPattern(pattern, timeField, fields) {
this.id = pattern;
this.title = pattern;
this.popularizeField = sinon.stub();
this.timeFieldName = timeField;
this.getNonScriptedFields = sinon.spy(IndexPattern.prototype.getNonScriptedFields);

4
src/ui/public/courier/data_source/build_query/__tests__/build_es_query.js
View file

@ -36,7 +36,7 @@ describe('build query', function () {
it('should combine queries and filters from multiple query languages into a single ES bool query', function () {
const queries = [
{ query: 'foo:bar', language: 'kuery' },
{ query: 'extension:jpg', language: 'kuery' },
{ query: 'bar:baz', language: 'lucene' },
];
const filters = [
@ -53,7 +53,7 @@ describe('build query', function () {
{ match_all: {} },
],
filter: [
toElasticsearchQuery(fromKueryExpression('foo:bar'), indexPattern),
toElasticsearchQuery(fromKueryExpression('extension:jpg'), indexPattern),
],
should: [],
must_not: [],

13
src/ui/public/courier/data_source/build_query/__tests__/from_kuery.js
View file

@ -2,6 +2,7 @@ import { buildQueryFromKuery } from '../from_kuery';
import StubbedLogstashIndexPatternProvider from 'fixtures/stubbed_logstash_index_pattern';
import ngMock from 'ng_mock';
import { expectDeepEqual } from '../../../../../../test_utils/expect_deep_equal.js';
import expect from 'expect.js';
import { fromKueryExpression, toElasticsearchQuery } from '../../../../kuery';
let indexPattern;
@ -28,8 +29,8 @@ describe('build query', function () {
it('should transform an array of kuery queries into ES queries combined in the bool\'s filter clause', function () {
const queries = [
{ query: 'foo:bar', language: 'kuery' },
{ query: 'bar:baz', language: 'kuery' },
{ query: 'extension:jpg', language: 'kuery' },
{ query: 'machine.os:osx', language: 'kuery' },
];
const expectedESQueries = queries.map(
@ -43,6 +44,14 @@ describe('build query', function () {
expectDeepEqual(result.filter, expectedESQueries);
});
it('should throw a useful error if it looks like query is using an old, unsupported syntax', function () {
const oldQuery = { query: 'is(foo, bar)', language: 'kuery' };
expect(buildQueryFromKuery).withArgs(indexPattern, [oldQuery]).to.throwError(
/It looks like you're using an outdated Kuery syntax./
);
});
});
});

11
src/ui/public/courier/data_source/build_query/build_es_query.js
View file

@ -1,6 +1,6 @@
import { groupBy, has } from 'lodash';
import { DecorateQueryProvider } from '../_decorate_query';
import { buildQueryFromKuery, buildQueryFromKql } from './from_kuery';
import { buildQueryFromKuery } from './from_kuery';
import { buildQueryFromFilters } from './from_filters';
import { buildQueryFromLucene } from './from_lucene';
@ -17,16 +17,15 @@ export function BuildESQueryProvider(Private) {
const queriesByLanguage = groupBy(validQueries, 'language');
const kueryQuery = buildQueryFromKuery(indexPattern, queriesByLanguage.kuery);
const kqlQuery = buildQueryFromKql(indexPattern, queriesByLanguage.kql);
const luceneQuery = buildQueryFromLucene(queriesByLanguage.lucene, decorateQuery);
const filterQuery = buildQueryFromFilters(filters, decorateQuery, indexPattern);
return {
bool: {
must: [].concat(kueryQuery.must, kqlQuery.must, luceneQuery.must, filterQuery.must),
filter: [].concat(kueryQuery.filter, kqlQuery.filter, luceneQuery.filter, filterQuery.filter),
should: [].concat(kueryQuery.should, kqlQuery.should, luceneQuery.should, filterQuery.should),
must_not: [].concat(kueryQuery.must_not, kqlQuery.must_not, luceneQuery.must_not, filterQuery.must_not),
must: [].concat(kueryQuery.must, luceneQuery.must, filterQuery.must),
filter: [].concat(kueryQuery.filter, luceneQuery.filter, filterQuery.filter),
should: [].concat(kueryQuery.should, luceneQuery.should, filterQuery.should),
must_not: [].concat(kueryQuery.must_not, luceneQuery.must_not, filterQuery.must_not),
}
};
}

28
src/ui/public/courier/data_source/build_query/from_kuery.js
View file

@ -1,13 +1,25 @@
import _ from 'lodash';
import { fromKueryExpression, fromKqlExpression, toElasticsearchQuery, nodeTypes } from '../../../kuery';
import { fromLegacyKueryExpression, fromKueryExpression, toElasticsearchQuery, nodeTypes } from '../../../kuery';
import { documentationLinks } from '../../../documentation_links';
export function buildQueryFromKuery(indexPattern, queries) {
const queryASTs = _.map(queries, query => fromKueryExpression(query.query));
return buildQuery(indexPattern, queryASTs);
}
const queryDocs = documentationLinks.query;
export function buildQueryFromKql(indexPattern, queries) {
const queryASTs = _.map(queries, query => fromKqlExpression(query.query));
export function buildQueryFromKuery(indexPattern, queries = []) {
const queryASTs = queries.map((query) => {
try {
return fromKueryExpression(query.query);
}
catch (parseError) {
try {
fromLegacyKueryExpression(query.query);
}
catch (legacyParseError) {
throw parseError;
}
throw new Error(
`It looks like you're using an outdated Kuery syntax. See what changed in the [docs](${queryDocs.kueryQuerySyntax})!`
);
}
});
return buildQuery(indexPattern, queryASTs);
}

32
src/ui/public/doc_table/__tests__/actions/filter.js
View file

@ -38,38 +38,6 @@ describe('doc table filter actions', function () {
expect(filterManager.add.calledWith(...args)).to.be(true);
});
it('should add an operator style "is" function to kuery queries', function () {
const state = {
query: { query: '', language: 'kuery' }
};
addFilter('foo', 'bar', '+', indexPattern, state, filterManager);
expect(state.query.query).to.be('"foo":"bar"');
});
it('should combine the new clause with any existing query clauses using an implicit "and"', function () {
const state = {
query: { query: 'foo', language: 'kuery' }
};
addFilter('foo', 'bar', '+', indexPattern, state, filterManager);
expect(state.query.query).to.be('foo "foo":"bar"');
});
it('should support creation of negated clauses', function () {
const state = {
query: { query: 'foo', language: 'kuery' }
};
addFilter('foo', 'bar', '-', indexPattern, state, filterManager);
expect(state.query.query).to.be('foo !"foo":"bar"');
});
it('should add an exists query when the provided field name is "_exists_"', function () {
const state = {
query: { query: 'foo', language: 'kuery' }
};
addFilter('_exists_', 'baz', '+', indexPattern, state, filterManager);
expect(state.query.query).to.be('foo exists("baz")');
});
});

31
src/ui/public/doc_table/actions/filter.js
View file

@ -1,36 +1,7 @@
import _ from 'lodash';
import { toKueryExpression, fromKueryExpression, nodeTypes } from 'ui/kuery';
export function addFilter(field, values = [], operation, index, state, filterManager) {
const fieldName = _.isObject(field) ? field.name : field;
if (!Array.isArray(values)) {
values = [values];
}
if (['lucene', 'kql'].includes(state.query.language)) {
filterManager.add(field, values, operation, index);
}
if (state.query.language === 'kuery') {
const negate = operation === '-';
const isExistsQuery = fieldName === '_exists_';
const newQueries = values.map((value) => {
const newQuery = isExistsQuery
? nodeTypes.function.buildNode('exists', value)
: nodeTypes.function.buildNode('is', fieldName, value);
return negate ? nodeTypes.function.buildNode('not', newQuery) : newQuery;
});
const allQueries = _.isEmpty(state.query.query)
? newQueries
: [fromKueryExpression(state.query.query), ...newQueries];
state.query = {
query: toKueryExpression(nodeTypes.function.buildNode('and', allQueries, 'implicit')),
language: 'kuery'
};
}
filterManager.add(field, values, operation, index);
}

16
src/ui/public/filter_bar/filter_bar_click_handler.js
View file

@ -3,10 +3,8 @@ import { dedupFilters } from './lib/dedup_filters';
import { uniqFilters } from './lib/uniq_filters';
import { findByParam } from 'ui/utils/find_by_param';
import { toastNotifications } from 'ui/notify';
import { AddFiltersToKueryProvider } from './lib/add_filters_to_kuery';
export function FilterBarClickHandlerProvider(Private) {
const addFiltersToKuery = Private(AddFiltersToKueryProvider);
export function FilterBarClickHandlerProvider() {
return function ($state) {
return function (event, simulate) {
@ -63,17 +61,7 @@ export function FilterBarClickHandlerProvider(Private) {
filters = dedupFilters($state.filters, uniqFilters(filters), { negate: true });
if (!simulate) {
if (['lucene', 'kql'].includes($state.query.language)) {
$state.$newFilters = filters;
}
else if ($state.query.language === 'kuery') {
addFiltersToKuery($state, filters)
.then(() => {
if (_.isFunction($state.save)) {
$state.save();
}
});
}
$state.$newFilters = filters;
}
return filters;
}

94
src/ui/public/filter_bar/lib/__tests__/add_filters_to_kuery.js
View file

@ -1,94 +0,0 @@
import { AddFiltersToKueryProvider } from '../add_filters_to_kuery';
import { FilterManagerProvider } from 'ui/filter_manager';
import NoDigestPromises from 'test_utils/no_digest_promises';
import expect from 'expect.js';
import ngMock from 'ng_mock';
import sinon from 'sinon';
import moment from 'moment';
describe('addFiltersToKuery', function () {
NoDigestPromises.activateForSuite();
let addFiltersToKuery;
let filterManager;
let timefilter;
beforeEach(ngMock.module(
'kibana',
'kibana/courier',
function ($provide) {
$provide.service('courier', require('fixtures/mock_courier'));
}
));
beforeEach(ngMock.inject(function (Private, _timefilter_) {
timefilter = _timefilter_;
addFiltersToKuery = Private(AddFiltersToKueryProvider);
filterManager = Private(FilterManagerProvider);
sinon.stub(filterManager, 'add');
}));
const filters = [{
meta: {
index: 'logstash-*',
type: 'phrase',
key: 'machine.os',
params: {
query: 'osx'
},
},
query: {
match: {
'machine.os': {
query: 'osx',
type: 'phrase'
}
}
}
}];
it('should return a Promise', function () {
const state = {
query: { query: '', language: 'lucene' }
};
expect(addFiltersToKuery(state, filters)).to.be.a(Promise);
});
it('should add a query clause equivalent to the given filter', function () {
const state = {
query: { query: '', language: 'kuery' }
};
return addFiltersToKuery(state, filters)
.then(() => {
expect(state.query.query).to.be('"machine.os":"osx"');
});
});
it('time field filters should update the global time filter instead of modifying the query', function () {
const startTime = moment('1995');
const endTime = moment('1996');
const state = {
query: { query: '', language: 'kuery' }
};
const timestampFilter = {
meta: {
index: 'logstash-*',
},
range: {
time: {
gt: startTime.valueOf(),
lt: endTime.valueOf(),
}
}
};
return addFiltersToKuery(state, [timestampFilter])
.then(() => {
expect(state.query.query).to.be('');
expect(startTime.isSame(timefilter.time.from)).to.be(true);
expect(endTime.isSame(timefilter.time.to)).to.be(true);
});
});
});

39
src/ui/public/filter_bar/lib/add_filters_to_kuery.js
View file

@ -1,39 +0,0 @@
import _ from 'lodash';
import { FilterBarLibMapAndFlattenFiltersProvider } from 'ui/filter_bar/lib/map_and_flatten_filters';
import { FilterBarLibExtractTimeFilterProvider } from 'ui/filter_bar/lib/extract_time_filter';
import { FilterBarLibChangeTimeFilterProvider } from 'ui/filter_bar/lib/change_time_filter';
import { FilterBarLibFilterOutTimeBasedFilterProvider } from 'ui/filter_bar/lib/filter_out_time_based_filter';
import { toKueryExpression, fromKueryExpression, nodeTypes, filterToKueryAST } from 'ui/kuery';
export function AddFiltersToKueryProvider(Private) {
const mapAndFlattenFilters = Private(FilterBarLibMapAndFlattenFiltersProvider);
const extractTimeFilter = Private(FilterBarLibExtractTimeFilterProvider);
const changeTimeFilter = Private(FilterBarLibChangeTimeFilterProvider);
const filterOutTimeBasedFilter = Private(FilterBarLibFilterOutTimeBasedFilterProvider);
return async function addFiltersToKuery(state, filters) {
return mapAndFlattenFilters(filters)
.then((results) => {
extractTimeFilter(results)
.then((timeFilter) => {
if (timeFilter) {
changeTimeFilter(timeFilter);
}
});
return results;
})
.then(filterOutTimeBasedFilter)
.then((results) => {
const newQueries = results.map(filterToKueryAST);
const allQueries = _.isEmpty(state.query.query)
? newQueries
: [fromKueryExpression(state.query.query), ...newQueries];
state.query = {
query: toKueryExpression(nodeTypes.function.buildNode('and', allQueries, 'implicit')),
language: 'kuery'
};
});
};
}

312
src/ui/public/kuery/ast/__tests__/ast.js
View file

@ -8,8 +8,8 @@ import { expectDeepEqual } from '../../../../../test_utils/expect_deep_equal.js'
// Helpful utility allowing us to test the PEG parser by simply checking for deep equality between
// the nodes the parser generates and the nodes our constructor functions generate.
function fromKueryExpressionNoMeta(text) {
return ast.fromKueryExpression(text, { includeMetadata: false });
function fromLegacyKueryExpressionNoMeta(text) {
return ast.fromLegacyKueryExpression(text, { includeMetadata: false });
}
let indexPattern;
@ -21,10 +21,10 @@ describe('kuery AST API', function () {
indexPattern = Private(StubbedLogstashIndexPatternProvider);
}));
describe('fromKueryExpression', function () {
describe('fromLegacyKueryExpression', function () {
it('should return location and text metadata for each AST node', function () {
const notNode = ast.fromKueryExpression('!foo:bar');
const notNode = ast.fromLegacyKueryExpression('!foo:bar');
expect(notNode).to.have.property('text', '!foo:bar');
expect(notNode.location).to.eql({ min: 0, max: 8 });
@ -42,19 +42,19 @@ describe('kuery AST API', function () {
it('should return a match all "is" function for whitespace', function () {
const expected = nodeTypes.function.buildNode('is', '*', '*');
const actual = fromKueryExpressionNoMeta(' ');
const actual = fromLegacyKueryExpressionNoMeta(' ');
expectDeepEqual(actual, expected);
});
it('should return an "and" function for single literals', function () {
const expected = nodeTypes.function.buildNode('and', [nodeTypes.literal.buildNode('foo')], 'implicit');
const actual = fromKueryExpressionNoMeta('foo');
const expected = nodeTypes.function.buildNode('and', [nodeTypes.literal.buildNode('foo')]);
const actual = fromLegacyKueryExpressionNoMeta('foo');
expectDeepEqual(actual, expected);
});
it('should ignore extraneous whitespace at the beginning and end of the query', function () {
const expected = nodeTypes.function.buildNode('and', [nodeTypes.literal.buildNode('foo')], 'implicit');
const actual = fromKueryExpressionNoMeta(' foo ');
const expected = nodeTypes.function.buildNode('and', [nodeTypes.literal.buildNode('foo')]);
const actual = fromLegacyKueryExpressionNoMeta(' foo ');
expectDeepEqual(actual, expected);
});
@ -62,8 +62,8 @@ describe('kuery AST API', function () {
const expected = nodeTypes.function.buildNode('and', [
nodeTypes.literal.buildNode('foo'),
nodeTypes.literal.buildNode('bar'),
], 'implicit');
const actual = fromKueryExpressionNoMeta('foo bar');
]);
const actual = fromLegacyKueryExpressionNoMeta('foo bar');
expectDeepEqual(actual, expected);
});
@ -71,8 +71,8 @@ describe('kuery AST API', function () {
const expected = nodeTypes.function.buildNode('and', [
nodeTypes.literal.buildNode('foo'),
nodeTypes.literal.buildNode('bar'),
], 'operator');
const actual = fromKueryExpressionNoMeta('foo and bar');
]);
const actual = fromLegacyKueryExpressionNoMeta('foo and bar');
expectDeepEqual(actual, expected);
});
@ -81,7 +81,7 @@ describe('kuery AST API', function () {
nodeTypes.literal.buildNode('foo'),
nodeTypes.literal.buildNode('bar'),
], 'function');
const actual = fromKueryExpressionNoMeta('and(foo, bar)');
const actual = fromLegacyKueryExpressionNoMeta('and(foo, bar)');
expectDeepEqual(actual, expected);
});
@ -89,8 +89,8 @@ describe('kuery AST API', function () {
const expected = nodeTypes.function.buildNode('or', [
nodeTypes.literal.buildNode('foo'),
nodeTypes.literal.buildNode('bar'),
], 'operator');
const actual = fromKueryExpressionNoMeta('foo or bar');
]);
const actual = fromLegacyKueryExpressionNoMeta('foo or bar');
expectDeepEqual(actual, expected);
});
@ -98,8 +98,8 @@ describe('kuery AST API', function () {
const expected = nodeTypes.function.buildNode('or', [
nodeTypes.literal.buildNode('foo'),
nodeTypes.literal.buildNode('bar'),
], 'function');
const actual = fromKueryExpressionNoMeta('or(foo, bar)');
]);
const actual = fromLegacyKueryExpressionNoMeta('or(foo, bar)');
expectDeepEqual(actual, expected);
});
@ -108,8 +108,8 @@ describe('kuery AST API', function () {
nodeTypes.function.buildNode('or', [
nodeTypes.literal.buildNode('foo'),
nodeTypes.literal.buildNode('bar'),
], 'function'), 'operator');
const actual = fromKueryExpressionNoMeta('!or(foo, bar)');
]));
const actual = fromLegacyKueryExpressionNoMeta('!or(foo, bar)');
expectDeepEqual(actual, expected);
});
@ -120,11 +120,11 @@ describe('kuery AST API', function () {
nodeTypes.function.buildNode('and', [
nodeTypes.literal.buildNode('bar'),
nodeTypes.literal.buildNode('baz'),
], 'operator'),
]),
nodeTypes.literal.buildNode('qux'),
])
], 'operator');
const actual = fromKueryExpressionNoMeta('foo or bar and baz or qux');
]);
const actual = fromLegacyKueryExpressionNoMeta('foo or bar and baz or qux');
expectDeepEqual(actual, expected);
});
@ -133,16 +133,16 @@ describe('kuery AST API', function () {
nodeTypes.function.buildNode('or', [
nodeTypes.literal.buildNode('foo'),
nodeTypes.literal.buildNode('bar'),
], 'operator'),
]),
nodeTypes.literal.buildNode('baz'),
], 'operator');
const actual = fromKueryExpressionNoMeta('(foo or bar) and baz');
]);
const actual = fromLegacyKueryExpressionNoMeta('(foo or bar) and baz');
expectDeepEqual(actual, expected);
});
it('should support a shorthand operator syntax for "is" functions', function () {
const expected = nodeTypes.function.buildNode('is', 'foo', 'bar', 'operator');
const actual = fromKueryExpressionNoMeta('foo:bar');
const expected = nodeTypes.function.buildNode('is', 'foo', 'bar', true);
const actual = fromLegacyKueryExpressionNoMeta('foo:bar');
expectDeepEqual(actual, expected);
});
@ -152,43 +152,223 @@ describe('kuery AST API', function () {
nodeTypes.literal.buildNode(1000),
nodeTypes.literal.buildNode(8000),
];
const expected = nodeTypes.function.buildNodeWithArgumentNodes('range', argumentNodes, 'operator');
const actual = fromKueryExpressionNoMeta('bytes:[1000 to 8000]');
const expected = nodeTypes.function.buildNodeWithArgumentNodes('range', argumentNodes);
const actual = fromLegacyKueryExpressionNoMeta('bytes:[1000 to 8000]');
expectDeepEqual(actual, expected);
});
it('should support functions with named arguments', function () {
const expected = nodeTypes.function.buildNode('range', 'bytes', { gt: 1000, lt: 8000 }, 'function');
const actual = fromKueryExpressionNoMeta('range(bytes, gt=1000, lt=8000)');
const expected = nodeTypes.function.buildNode('range', 'bytes', { gt: 1000, lt: 8000 });
const actual = fromLegacyKueryExpressionNoMeta('range(bytes, gt=1000, lt=8000)');
expectDeepEqual(actual, expected);
});
it('should throw an error for unknown functions', function () {
expect(ast.fromKueryExpression).withArgs('foo(bar)').to.throwException(/Unknown function "foo"/);
expect(ast.fromLegacyKueryExpression).withArgs('foo(bar)').to.throwException(/Unknown function "foo"/);
});
});
describe('toKueryExpression', function () {
describe('fromKueryExpression', function () {
it('should return the given node type\'s kuery string representation', function () {
const node = nodeTypes.function.buildNode('exists', 'foo');
const expected = nodeTypes.function.toKueryExpression(node);
const result = ast.toKueryExpression(node);
expectDeepEqual(result, expected);
it('should return a match all "is" function for whitespace', function () {
const expected = nodeTypes.function.buildNode('is', '*', '*');
const actual = ast.fromKueryExpression(' ');
expectDeepEqual(actual, expected);
});
it('should return an empty string for undefined nodes and unknown node types', function () {
expect(ast.toKueryExpression()).to.be('');
const noTypeNode = nodeTypes.function.buildNode('exists', 'foo');
delete noTypeNode.type;
expect(ast.toKueryExpression(noTypeNode)).to.be('');
const unknownTypeNode = nodeTypes.function.buildNode('exists', 'foo');
unknownTypeNode.type = 'notValid';
expect(ast.toKueryExpression(unknownTypeNode)).to.be('');
it('should return an "is" function with a null field for single literals', function () {
const expected = nodeTypes.function.buildNode('is', null, 'foo');
const actual = ast.fromKueryExpression('foo');
expectDeepEqual(actual, expected);
});
it('should ignore extraneous whitespace at the beginning and end of the query', function () {
const expected = nodeTypes.function.buildNode('is', null, 'foo');
const actual = ast.fromKueryExpression(' foo ');
expectDeepEqual(actual, expected);
});
it('should not split on whitespace', function () {
const expected = nodeTypes.function.buildNode('is', null, 'foo bar');
const actual = ast.fromKueryExpression('foo bar');
expectDeepEqual(actual, expected);
});
it('should support "and" as a binary operator', function () {
const expected = nodeTypes.function.buildNode('and', [
nodeTypes.function.buildNode('is', null, 'foo'),
nodeTypes.function.buildNode('is', null, 'bar'),
]);
const actual = ast.fromKueryExpression('foo and bar');
expectDeepEqual(actual, expected);
});
it('should support "or" as a binary operator', function () {
const expected = nodeTypes.function.buildNode('or', [
nodeTypes.function.buildNode('is', null, 'foo'),
nodeTypes.function.buildNode('is', null, 'bar'),
]);
const actual = ast.fromKueryExpression('foo or bar');
expectDeepEqual(actual, expected);
});
it('should support negation of queries with a "not" prefix', function () {
const expected = nodeTypes.function.buildNode('not',
nodeTypes.function.buildNode('or', [
nodeTypes.function.buildNode('is', null, 'foo'),
nodeTypes.function.buildNode('is', null, 'bar'),
])
);
const actual = ast.fromKueryExpression('not (foo or bar)');
expectDeepEqual(actual, expected);
});
it('"and" should have a higher precedence than "or"', function () {
const expected = nodeTypes.function.buildNode('or', [
nodeTypes.function.buildNode('is', null, 'foo'),
nodeTypes.function.buildNode('or', [
nodeTypes.function.buildNode('and', [
nodeTypes.function.buildNode('is', null, 'bar'),
nodeTypes.function.buildNode('is', null, 'baz'),
]),
nodeTypes.function.buildNode('is', null, 'qux'),
])
]);
const actual = ast.fromKueryExpression('foo or bar and baz or qux');
expectDeepEqual(actual, expected);
});
it('should support grouping to override default precedence', function () {
const expected = nodeTypes.function.buildNode('and', [
nodeTypes.function.buildNode('or', [
nodeTypes.function.buildNode('is', null, 'foo'),
nodeTypes.function.buildNode('is', null, 'bar'),
]),
nodeTypes.function.buildNode('is', null, 'baz'),
]);
const actual = ast.fromKueryExpression('(foo or bar) and baz');
expectDeepEqual(actual, expected);
});
it('should support matching against specific fields', function () {
const expected = nodeTypes.function.buildNode('is', 'foo', 'bar');
const actual = ast.fromKueryExpression('foo:bar');
expectDeepEqual(actual, expected);
});
it('should also not split on whitespace when matching specific fields', function () {
const expected = nodeTypes.function.buildNode('is', 'foo', 'bar baz');
const actual = ast.fromKueryExpression('foo:bar baz');
expectDeepEqual(actual, expected);
});
it('should treat quoted values as phrases', function () {
const expected = nodeTypes.function.buildNode('is', 'foo', 'bar baz', true);
const actual = ast.fromKueryExpression('foo:"bar baz"');
expectDeepEqual(actual, expected);
});
it('should support a shorthand for matching multiple values against a single field', function () {
const expected = nodeTypes.function.buildNode('or', [
nodeTypes.function.buildNode('is', 'foo', 'bar'),
nodeTypes.function.buildNode('is', 'foo', 'baz'),
]);
const actual = ast.fromKueryExpression('foo:(bar or baz)');
expectDeepEqual(actual, expected);
});
it('should support "and" and "not" operators and grouping in the shorthand as well', function () {
const expected = nodeTypes.function.buildNode('and', [
nodeTypes.function.buildNode('or', [
nodeTypes.function.buildNode('is', 'foo', 'bar'),
nodeTypes.function.buildNode('is', 'foo', 'baz'),
]),
nodeTypes.function.buildNode('not',
nodeTypes.function.buildNode('is', 'foo', 'qux')
),
]);
const actual = ast.fromKueryExpression('foo:((bar or baz) and not qux)');
expectDeepEqual(actual, expected);
});
it('should support exclusive range operators', function () {
const expected = nodeTypes.function.buildNode('and', [
nodeTypes.function.buildNode('range', 'bytes', {
gt: 1000,
}),
nodeTypes.function.buildNode('range', 'bytes', {
lt: 8000,
}),
]);
const actual = ast.fromKueryExpression('bytes > 1000 and bytes < 8000');
expectDeepEqual(actual, expected);
});
it('should support inclusive range operators', function () {
const expected = nodeTypes.function.buildNode('and', [
nodeTypes.function.buildNode('range', 'bytes', {
gte: 1000,
}),
nodeTypes.function.buildNode('range', 'bytes', {
lte: 8000,
}),
]);
const actual = ast.fromKueryExpression('bytes >= 1000 and bytes <= 8000');
expectDeepEqual(actual, expected);
});
it('should support wildcards in field names', function () {
const expected = nodeTypes.function.buildNode('is', 'machine*', 'osx');
const actual = ast.fromKueryExpression('machine*:osx');
expectDeepEqual(actual, expected);
});
it('should support wildcards in values', function () {
const expected = nodeTypes.function.buildNode('is', 'foo', 'ba*');
const actual = ast.fromKueryExpression('foo:ba*');
expectDeepEqual(actual, expected);
});
it('should create an exists "is" query when a field is given and "*" is the value', function () {
const expected = nodeTypes.function.buildNode('is', 'foo', '*');
const actual = ast.fromKueryExpression('foo:*');
expectDeepEqual(actual, expected);
});
});
describe('fromLiteralExpression', function () {
it('should create literal nodes for unquoted values with correct primitive types', function () {
const stringLiteral = nodeTypes.literal.buildNode('foo');
const booleanFalseLiteral = nodeTypes.literal.buildNode(false);
const booleanTrueLiteral = nodeTypes.literal.buildNode(true);
const numberLiteral = nodeTypes.literal.buildNode(42);
expectDeepEqual(ast.fromLiteralExpression('foo'), stringLiteral);
expectDeepEqual(ast.fromLiteralExpression('true'), booleanTrueLiteral);
expectDeepEqual(ast.fromLiteralExpression('false'), booleanFalseLiteral);
expectDeepEqual(ast.fromLiteralExpression('42'), numberLiteral);
});
it('should allow escaping of special characters with a backslash', function () {
const expected = nodeTypes.literal.buildNode('\\():<>"*');
// yo dawg
const actual = ast.fromLiteralExpression('\\\\\\(\\)\\:\\<\\>\\"\\*');
expectDeepEqual(actual, expected);
});
it('should support double quoted strings that do not need escapes except for quotes', function () {
const expected = nodeTypes.literal.buildNode('\\():<>"*');
const actual = ast.fromLiteralExpression('"\\():<>\\"*"');
expectDeepEqual(actual, expected);
});
it('should detect wildcards and build wildcard AST nodes', function () {
const expected = nodeTypes.wildcard.buildNode('foo*bar');
const actual = ast.fromLiteralExpression('foo*bar');
expectDeepEqual(actual, expected);
});
});
describe('toElasticsearchQuery', function () {
@ -216,38 +396,4 @@ describe('kuery AST API', function () {
});
describe('symmetry of to/fromKueryExpression', function () {
it('toKueryExpression and fromKueryExpression should be inverse operations', function () {
function testExpression(expression) {
expect(ast.toKueryExpression(ast.fromKueryExpression(expression))).to.be(expression);
}
testExpression('');
testExpression(' ');
testExpression('foo');
testExpression('foo bar');
testExpression('foo 200');
testExpression('bytes:[1000 to 8000]');
testExpression('bytes:[1000 TO 8000]');
testExpression('range(bytes, gt=1000, lt=8000)');
testExpression('range(bytes, gt=1000, lte=8000)');
testExpression('range(bytes, gte=1000, lt=8000)');
testExpression('range(bytes, gte=1000, lte=8000)');
testExpression('response:200');
testExpression('"response":200');
testExpression('response:"200"');
testExpression('"response":"200"');
testExpression('is(response, 200)');
testExpression('!is(response, 200)');
testExpression('foo or is(tic, tock) or foo:bar');
testExpression('or(foo, is(tic, tock), foo:bar)');
testExpression('foo is(tic, tock) foo:bar');
testExpression('foo and is(tic, tock) and foo:bar');
testExpression('(foo or is(tic, tock)) and foo:bar');
testExpression('!(foo or is(tic, tock)) and foo:bar');
});
});
});

37
src/ui/public/kuery/ast/ast.js
View file

@ -1,21 +1,32 @@
import grammar from 'raw-loader!./kuery.peg';
import kqlGrammar from 'raw-loader!./kql.peg';
import legacyKueryGrammar from 'raw-loader!./legacy_kuery.peg';
import kueryGrammar from 'raw-loader!./kuery.peg';
import PEG from 'pegjs';
import _ from 'lodash';
import { nodeTypes } from '../node_types/index';
const kueryParser = PEG.buildParser(grammar);
const kqlParser = PEG.buildParser(kqlGrammar);
const legacyKueryParser = PEG.buildParser(legacyKueryGrammar);
const kueryParser = PEG.buildParser(kueryGrammar, {
allowedStartRules: ['start', 'Literal'],
});
export function fromLiteralExpression(expression, parseOptions) {
parseOptions = {
...parseOptions,
startRule: 'Literal',
};
return fromExpression(expression, parseOptions, kueryParser);
}
export function fromLegacyKueryExpression(expression, parseOptions) {
return fromExpression(expression, parseOptions, legacyKueryParser);
}
export function fromKueryExpression(expression, parseOptions) {
return fromExpression(expression, parseOptions, kueryParser);
}
export function fromKqlExpression(expression, parseOptions) {
return fromExpression(expression, parseOptions, kqlParser);
}
function fromExpression(expression, parseOptions = {}, parser = kqlParser) {
function fromExpression(expression, parseOptions = {}, parser = kueryParser) {
if (_.isUndefined(expression)) {
throw new Error('expression must be a string, got undefined instead');
}
@ -28,14 +39,6 @@ function fromExpression(expression, parseOptions = {}, parser = kqlParser) {
return parser.parse(expression, parseOptions);
}
export function toKueryExpression(node) {
if (!node || !node.type || !nodeTypes[node.type]) {
return '';
}
return nodeTypes[node.type].toKueryExpression(node);
}
export function toElasticsearchQuery(node, indexPattern) {
if (!node || !node.type || !nodeTypes[node.type]) {
return toElasticsearchQuery(nodeTypes.function.buildNode('and', []));

2
src/ui/public/kuery/ast/index.js
View file

@ -1 +1 @@
export { fromKueryExpression, fromKqlExpression, toKueryExpression, toElasticsearchQuery } from './ast';
export { fromLegacyKueryExpression, fromKueryExpression, fromLiteralExpression, toElasticsearchQuery } from './ast';

279
src/ui/public/kuery/ast/kuery.peg
View file

@ -1,150 +1,179 @@
/*
* Kuery parser
*/
/*
* Initialization block
*/
// Initialization block
{
var nodeTypes = options.helpers.nodeTypes;
const { nodeTypes } = options.helpers;
const buildFunctionNode = nodeTypes.function.buildNodeWithArgumentNodes;
const buildLiteralNode = nodeTypes.literal.buildNode;
const buildWildcardNode = nodeTypes.wildcard.buildNode;
const buildNamedArgNode = nodeTypes.namedArg.buildNode;
if (options.includeMetadata === undefined) {
options.includeMetadata = true;
function trimLeft(string) {
return string.replace(/^[\s\uFEFF\xA0]+/g, '');
}
function addMeta(source, text, location) {
if (options.includeMetadata) {
return Object.assign(
{},
source,
{
text: text,
location: simpleLocation(location),
}
);
}
return source;
}
function simpleLocation(location) {
// Returns an object representing the position of the function within the expression,
// demarcated by the position of its first character and last character. We calculate these values
// using the offset because the expression could span multiple lines, and we don't want to deal
// with column and line values.
return {
min: location.start.offset,
max: location.end.offset
}
function trimRight(string) {
return string.replace(/[\s\uFEFF\xA0]+$/g, '');
}
}
start
= space? query:OrQuery space? {
if (query.type === 'literal') {
return addMeta(nodeTypes.function.buildNode('and', [query], 'implicit'), text(), location());
}
return query;
}
/ whitespace:[\ \t\r\n]* {
return addMeta(nodeTypes.function.buildNode('is', '*', '*'), text(), location());
= Space* query:OrQuery? Space* {
if (query !== null) return query;
return nodeTypes.function.buildNode('is', '*', '*');
}
OrQuery
= left:AndQuery space 'or'i space right:OrQuery {
return addMeta(nodeTypes.function.buildNode('or', [left, right], 'operator'), text(), location());
= left:AndQuery Or right:OrQuery {
return buildFunctionNode('or', [left, right]);
}
/ AndQuery
AndQuery
= left:NegatedClause space 'and'i space right:AndQuery {
return addMeta(nodeTypes.function.buildNode('and', [left, right], 'operator'), text(), location());
= left:NotQuery And right:AndQuery{
return buildFunctionNode('and', [left, right]);
}
/ left:NegatedClause space !'or'i right:AndQuery {
return addMeta(nodeTypes.function.buildNode('and', [left, right], 'implicit'), text(), location());
/ NotQuery
NotQuery
= Not query:SubQuery {
return buildFunctionNode('not', [query]);
}
/ NegatedClause
/ SubQuery
NegatedClause
= [!] clause:Clause {
return addMeta(nodeTypes.function.buildNode('not', clause, 'operator'), text(), location());
}
/ Clause
SubQuery
= '(' Space* query:OrQuery Space* ')' { return query; }
/ Expression
Clause
= '(' subQuery:start ')' {
return subQuery;
}
/ Term
Expression
= FieldRangeExpression
/ FieldValueExpression
/ ValueExpression
Term
= field:literal_arg_type ':' value:literal_arg_type {
return addMeta(nodeTypes.function.buildNodeWithArgumentNodes('is', [field, value], 'operator'), text(), location());
}
/ field:literal_arg_type ':[' space? gt:literal_arg_type space 'to'i space lt:literal_arg_type space? ']' {
return addMeta(nodeTypes.function.buildNodeWithArgumentNodes('range', [field, gt, lt], 'operator'), text(), location());
}
/ function
/ !Keywords literal:literal_arg_type { return literal; }
function_name
= first:[a-zA-Z]+ rest:[.a-zA-Z0-9_-]* { return first.join('') + rest.join('') }
function "function"
= name:function_name space? '(' space? arg_list:arg_list? space? ')' {
return addMeta(nodeTypes.function.buildNodeWithArgumentNodes(name, arg_list || [], 'function'), text(), location());
}
arg_list
= first:argument rest:(space? ',' space? arg:argument {return arg})* space? ','? {
return [first].concat(rest);
}
argument
= name:function_name space? '=' space? value:arg_type {
return addMeta(nodeTypes.namedArg.buildNode(name, value), text(), location());
}
/ element:arg_type {return element}
arg_type
= OrQuery
/ literal_arg_type
literal_arg_type
= literal:literal {
var result = addMeta(nodeTypes.literal.buildNode(literal), text(), location());
return result;
}
Keywords
= 'and'i / 'or'i
/* ----- Core types ----- */
literal "literal"
= '"' chars:dq_char* '"' { return chars.join(''); } // double quoted string
/ "'" chars:sq_char* "'" { return chars.join(''); } // single quoted string
/ 'true' { return true; } // unquoted literals from here down
/ 'false' { return false; }
/ 'null' { return null; }
/ string:[^\[\]()"',:=\ \t]+ { // this also matches numbers via Number()
var result = string.join('');
// Sort of hacky, but PEG doesn't have backtracking so
// a number rule is hard to read, and performs worse
if (isNaN(Number(result))) return result;
return Number(result)
FieldRangeExpression
= field:Literal Space* operator:RangeOperator Space* value:(QuotedString / UnquotedLiteral) {
const range = buildNamedArgNode(operator, value);
return buildFunctionNode('range', [field, range]);
}
space
= [\ \t\r\n]+
FieldValueExpression
= field:Literal Space* ':' Space* partial:ListOfValues {
return partial(field);
}
dq_char
= "\\" sequence:('"' / "\\") { return sequence; }
/ [^"] // everything except "
ValueExpression
= partial:Value {
const field = buildLiteralNode(null);
return partial(field);
}
sq_char
= "\\" sequence:("'" / "\\") { return sequence; }
/ [^'] // everything except '
ListOfValues
= '(' Space* partial:OrListOfValues Space* ')' { return partial; }
/ Value
integer
= digits:[0-9]+ {return parseInt(digits.join(''))}
OrListOfValues
= partialLeft:AndListOfValues Or partialRight:OrListOfValues {
return (field) => buildFunctionNode('or', [partialLeft(field), partialRight(field)]);
}
/ AndListOfValues
AndListOfValues
= partialLeft:NotListOfValues And partialRight:AndListOfValues {
return (field) => buildFunctionNode('and', [partialLeft(field), partialRight(field)]);
}
/ NotListOfValues
NotListOfValues
= Not partial:ListOfValues {
return (field) => buildFunctionNode('not', [partial(field)]);
}
/ ListOfValues
Value
= value:QuotedString {
const isPhrase = buildLiteralNode(true);
return (field) => buildFunctionNode('is', [field, value, isPhrase]);
}
/ value:WildcardString {
const isPhrase = buildLiteralNode(false);
return (field) => buildFunctionNode('is', [field, value, isPhrase]);
}
/ value:UnquotedLiteral {
const isPhrase = buildLiteralNode(false);
return (field) => buildFunctionNode('is', [field, value, isPhrase]);
}
Or
= Space+ 'or'i Space+
And
= Space+ 'and'i Space+
Not
= 'not'i Space+
Literal
= QuotedString / WildcardString / UnquotedLiteral
QuotedString
= '"' chars:(EscapedDoubleQuote / [^"])* '"' {
return buildLiteralNode(chars.join(''));
}
WildcardString
= sequences:WildcardSequence+ {
const compactedSequences = sequences.reduce((acc, arr, i) => {
const compacted = arr.filter(value => value !== '');
return [...acc, ...compacted];
}, []);
if (typeof compactedSequences[0] === 'string') {
compactedSequences[0] = trimLeft(compactedSequences[0]);
}
const lastIndex = compactedSequences.length - 1;
if (typeof compactedSequences[lastIndex] === 'string') {
compactedSequences[lastIndex] = trimRight(compactedSequences[lastIndex]);
}
return buildWildcardNode(compactedSequences);
}
WildcardSequence
= left:UnquotedCharacter* '*' right:UnquotedCharacter* {
return [left.join(''), nodeTypes.wildcard.wildcardSymbol, right.join('')];
}
UnquotedLiteral
= chars:UnquotedCharacter+ {
const sequence = chars.join('').trim();
if (sequence === 'null') return buildLiteralNode(null);
if (sequence === 'true') return buildLiteralNode(true);
if (sequence === 'false') return buildLiteralNode(false);
const number = Number(sequence);
const value = isNaN(number) ? sequence : number;
return buildLiteralNode(value);
}
UnquotedCharacter
= EscapedSpecialCharacter
/ !Separator char:. { return char; }
EscapedSpecialCharacter
= '\\' char:SpecialCharacter { return char; }
EscapedDoubleQuote
= '\\' char:'"' { return char; }
Separator
= Keyword / SpecialCharacter
Keyword
= Or / And / Not
SpecialCharacter
= [\\():<>"*]
RangeOperator
= '<=' { return 'lte'; }
/ '>=' { return 'gte'; }
/ '<' { return 'lt'; }
/ '>' { return 'gt'; }
Space
= [\ \t\r\n]

61
src/ui/public/kuery/ast/kql.peg → src/ui/public/kuery/ast/legacy_kuery.peg
View file

@ -40,18 +40,15 @@
}
start
= Query
/ space* {
return addMeta(nodeTypes.function.buildNode('and', []), text(), location());
}
Query
= space? query:OrQuery space? {
if (query.type === 'literal') {
return addMeta(nodeTypes.function.buildNode('and', [query]), text(), location());
}
return query;
}
/ whitespace:[\ \t\r\n]* {
return addMeta(nodeTypes.function.buildNode('is', '*', '*', false), text(), location());
}
OrQuery
= left:AndQuery space 'or'i space right:OrQuery {
@ -60,40 +57,67 @@ OrQuery
/ AndQuery
AndQuery
= left:NotQuery space 'and'i space right:AndQuery {
= left:NegatedClause space 'and'i space right:AndQuery {
return addMeta(nodeTypes.function.buildNode('and', [left, right]), text(), location());
}
/ NotQuery
/ left:NegatedClause space !'or'i right:AndQuery {
return addMeta(nodeTypes.function.buildNode('and', [left, right]), text(), location());
}
/ NegatedClause
NotQuery
= 'not'i space clause:Clause {
NegatedClause
= [!] clause:Clause {
return addMeta(nodeTypes.function.buildNode('not', clause), text(), location());
}
/ Clause
Clause
= '(' subQuery:Query ')' {
= '(' subQuery:start ')' {
return subQuery;
}
/ Term
Term
= field:literal_arg_type space? ':' space? value:literal_arg_type {
return addMeta(nodeTypes.function.buildNodeWithArgumentNodes('is', [field, value]), text(), location());
= field:literal_arg_type ':' value:literal_arg_type {
return addMeta(nodeTypes.function.buildNodeWithArgumentNodes('is', [field, value, nodeTypes.literal.buildNode(true)]), text(), location());
}
/ field:literal_arg_type space? ':' space? '[' space? gt:literal_arg_type space 'to'i space lt:literal_arg_type space? ']' {
/ field:literal_arg_type ':[' space? gt:literal_arg_type space 'to'i space lt:literal_arg_type space? ']' {
return addMeta(nodeTypes.function.buildNodeWithArgumentNodes('range', [field, gt, lt]), text(), location());
}
/ function
/ !Keywords literal:literal_arg_type { return literal; }
function_name
= first:[a-zA-Z]+ rest:[.a-zA-Z0-9_-]* { return first.join('') + rest.join('') }
function "function"
= name:function_name space? '(' space? arg_list:arg_list? space? ')' {
return addMeta(nodeTypes.function.buildNodeWithArgumentNodes(name, arg_list || []), text(), location());
}
arg_list
= first:argument rest:(space? ',' space? arg:argument {return arg})* space? ','? {
return [first].concat(rest);
}
argument
= name:function_name space? '=' space? value:arg_type {
return addMeta(nodeTypes.namedArg.buildNode(name, value), text(), location());
}
/ element:arg_type {return element}
arg_type
= OrQuery
/ literal_arg_type
literal_arg_type
= literal:literal {
var result = addMeta(nodeTypes.literal.buildNode(literal), text(), location());
return result;
var result = addMeta(nodeTypes.literal.buildNode(literal), text(), location());
return result;
}
Keywords
= 'or'i / 'and'i / 'not'i
= 'and'i / 'or'i
/* ----- Core types ----- */
@ -121,3 +145,6 @@ dq_char
sq_char
= "\\" sequence:("'" / "\\") { return sequence; }
/ [^'] // everything except '
integer
= digits:[0-9]+ {return parseInt(digits.join(''))}

52
src/ui/public/kuery/functions/__tests__/and.js
View file

@ -4,11 +4,10 @@ import { nodeTypes } from '../../node_types';
import * as ast from '../../ast';
import StubbedLogstashIndexPatternProvider from 'fixtures/stubbed_logstash_index_pattern';
import ngMock from 'ng_mock';
import { expectDeepEqual } from '../../../../../test_utils/expect_deep_equal';
let indexPattern;
const childNode1 = nodeTypes.function.buildNode('is', 'response', 200);
const childNode1 = nodeTypes.function.buildNode('is', 'machine.os', 'osx');
const childNode2 = nodeTypes.function.buildNode('is', 'extension', 'jpg');
describe('kuery functions', function () {
@ -22,11 +21,6 @@ describe('kuery functions', function () {
describe('buildNodeParams', function () {
it('should return "arguments" and "serializeStyle" params', function () {
const result = and.buildNodeParams([childNode1, childNode2]);
expect(result).to.only.have.keys('arguments', 'serializeStyle');
});
it('arguments should contain the unmodified child nodes', function () {
const result = and.buildNodeParams([childNode1, childNode2]);
const { arguments: [ actualChildNode1, actualChildNode2 ] } = result;
@ -34,11 +28,6 @@ describe('kuery functions', function () {
expect(actualChildNode2).to.be(childNode2);
});
it('serializeStyle should default to "operator"', function () {
const { serializeStyle } = and.buildNodeParams([childNode1, childNode2]);
expect(serializeStyle).to.be('operator');
});
});
describe('toElasticsearchQuery', function () {
@ -53,46 +42,7 @@ describe('kuery functions', function () {
);
});
it('should wrap a literal argument with an "is" function targeting the default_field', function () {
const literalFoo = nodeTypes.literal.buildNode('foo');
const expectedChild = ast.toElasticsearchQuery(nodeTypes.function.buildNode('is', null, 'foo'), indexPattern);
const node = nodeTypes.function.buildNode('and', [literalFoo]);
const result = and.toElasticsearchQuery(node, indexPattern);
const resultChild = result.bool.filter[0];
expectDeepEqual(resultChild, expectedChild);
});
});
describe('toKueryExpression', function () {
it('should serialize "and" nodes with an implicit syntax when requested', function () {
const node = nodeTypes.function.buildNode('and', [childNode1, childNode2], 'implicit');
const result = and.toKueryExpression(node);
expect(result).to.be('"response":200 "extension":"jpg"');
});
it('should serialize "and" nodes with an operator syntax when requested', function () {
const node = nodeTypes.function.buildNode('and', [childNode1, childNode2], 'operator');
const result = and.toKueryExpression(node);
expect(result).to.be('"response":200 and "extension":"jpg"');
});
it('should wrap "or" sub-queries in parenthesis', function () {
const orNode = nodeTypes.function.buildNode('or', [childNode1, childNode2], 'operator');
const fooBarNode = nodeTypes.function.buildNode('is', 'foo', 'bar');
const andNode = nodeTypes.function.buildNode('and', [orNode, fooBarNode], 'implicit');
const result = and.toKueryExpression(andNode);
expect(result).to.be('("response":200 or "extension":"jpg") "foo":"bar"');
});
it('should throw an error for nodes with unknown or undefined serialize styles', function () {
const node = nodeTypes.function.buildNode('and', [childNode1, childNode2], 'notValid');
expect(and.toKueryExpression)
.withArgs(node).to.throwException(/Cannot serialize "and" function as "notValid"/);
});
});
});
});

113
src/ui/public/kuery/functions/__tests__/is.js
View file

@ -23,13 +23,8 @@ describe('kuery functions', function () {
expect(is.buildNodeParams).withArgs('foo').to.throwException(/value is a required argument/);
});
it('should return "arguments" and "serializeStyle" params', function () {
const result = is.buildNodeParams('response', 200);
expect(result).to.only.have.keys('arguments', 'serializeStyle');
});
it('arguments should contain the provided fieldName and value as literals', function () {
const { arguments: [ fieldName, value ] } = is.buildNodeParams('response', 200);
const { arguments: [fieldName, value] } = is.buildNodeParams('response', 200);
expect(fieldName).to.have.property('type', 'literal');
expect(fieldName).to.have.property('value', 'response');
@ -38,11 +33,22 @@ describe('kuery functions', function () {
expect(value).to.have.property('value', 200);
});
it('serializeStyle should default to "operator"', function () {
const { serializeStyle } = is.buildNodeParams('response', 200);
expect(serializeStyle).to.be('operator');
it('should detect wildcards in the provided arguments', function () {
const { arguments: [fieldName, value] } = is.buildNodeParams('machine*', 'win*');
expect(fieldName).to.have.property('type', 'wildcard');
expect(value).to.have.property('type', 'wildcard');
});
it('should default to a non-phrase query', function () {
const { arguments: [, , isPhrase] } = is.buildNodeParams('response', 200);
expect(isPhrase.value).to.be(false);
});
it('should allow specification of a phrase query', function () {
const { arguments: [, , isPhrase] } = is.buildNodeParams('response', 200, true);
expect(isPhrase.value).to.be(true);
});
});
describe('toElasticsearchQuery', function () {
@ -61,7 +67,7 @@ describe('kuery functions', function () {
const expected = {
multi_match: {
query: 200,
type: 'phrase',
type: 'best_fields',
lenient: true,
}
};
@ -71,39 +77,74 @@ describe('kuery functions', function () {
expectDeepEqual(result, expected);
});
it('should return an ES multi_match query when fieldName is "*"', function () {
const expected = {
multi_match: {
query: 200,
fields: ['*'],
type: 'phrase',
lenient: true,
}
};
it('should return an ES bool query with a sub-query for each field when fieldName is "*"', function () {
const node = nodeTypes.function.buildNode('is', '*', 200);
const result = is.toElasticsearchQuery(node, indexPattern);
expectDeepEqual(result, expected);
expect(result).to.have.property('bool');
expect(result.bool.should).to.have.length(indexPattern.fields.length);
});
it('should return an ES exists query when value is "*"', function () {
const expected = {
exists: { field: 'response' }
bool: {
should: [
{ exists: { field: 'extension' } },
],
minimum_should_match: 1
}
};
const node = nodeTypes.function.buildNode('is', 'response', '*');
const node = nodeTypes.function.buildNode('is', 'extension', '*');
const result = is.toElasticsearchQuery(node, indexPattern);
expectDeepEqual(result, expected);
});
it('should return an ES match_phrase query when a concrete fieldName and value are provided', function () {
it('should return an ES match query when a concrete fieldName and value are provided', function () {
const expected = {
match_phrase: {
response: 200
bool: {
should: [
{ match: { extension: 'jpg' } },
],
minimum_should_match: 1
}
};
const node = nodeTypes.function.buildNode('is', 'response', 200);
const node = nodeTypes.function.buildNode('is', 'extension', 'jpg');
const result = is.toElasticsearchQuery(node, indexPattern);
expectDeepEqual(result, expected);
});
it('should support creation of phrase queries', function () {
const expected = {
bool: {
should: [
{ match_phrase: { extension: 'jpg' } },
],
minimum_should_match: 1
}
};
const node = nodeTypes.function.buildNode('is', 'extension', 'jpg', true);
const result = is.toElasticsearchQuery(node, indexPattern);
expectDeepEqual(result, expected);
});
it('should create a query_string query for wildcard values', function () {
const expected = {
bool: {
should: [
{
query_string: {
fields: ['extension'],
query: 'jpg*'
}
},
],
minimum_should_match: 1
}
};
const node = nodeTypes.function.buildNode('is', 'extension', 'jpg*');
const result = is.toElasticsearchQuery(node, indexPattern);
expectDeepEqual(result, expected);
});
@ -111,23 +152,7 @@ describe('kuery functions', function () {
it('should support scripted fields', function () {
const node = nodeTypes.function.buildNode('is', 'script string', 'foo');
const result = is.toElasticsearchQuery(node, indexPattern);
expect(result).to.have.key('script');
});
});
describe('toKueryExpression', function () {
it('should serialize "is" nodes with an operator syntax', function () {
const node = nodeTypes.function.buildNode('is', 'response', 200, 'operator');
const result = is.toKueryExpression(node);
expect(result).to.be('"response":200');
});
it('should throw an error for nodes with unknown or undefined serialize styles', function () {
const node = nodeTypes.function.buildNode('is', 'response', 200, 'notValid');
expect(is.toKueryExpression)
.withArgs(node).to.throwException(/Cannot serialize "is" function as "notValid"/);
expect(result.bool.should[0]).to.have.key('script');
});
});

58
src/ui/public/kuery/functions/__tests__/not.js
View file

@ -4,11 +4,10 @@ import { nodeTypes } from '../../node_types';
import * as ast from '../../ast';
import StubbedLogstashIndexPatternProvider from 'fixtures/stubbed_logstash_index_pattern';
import ngMock from 'ng_mock';
import { expectDeepEqual } from '../../../../../test_utils/expect_deep_equal';
let indexPattern;
const childNode = nodeTypes.function.buildNode('is', 'response', 200);
const childNode = nodeTypes.function.buildNode('is', 'extension', 'jpg');
describe('kuery functions', function () {
@ -21,20 +20,11 @@ describe('kuery functions', function () {
describe('buildNodeParams', function () {
it('should return "arguments" and "serializeStyle" params', function () {
const result = not.buildNodeParams(childNode);
expect(result).to.only.have.keys('arguments', 'serializeStyle');
});
it('arguments should contain the unmodified child node', function () {
const { arguments: [ actualChild ] } = not.buildNodeParams(childNode);
expect(actualChild).to.be(childNode);
});
it('serializeStyle should default to "operator"', function () {
const { serializeStyle } = not.buildNodeParams(childNode);
expect(serializeStyle).to.be('operator');
});
});
@ -47,52 +37,6 @@ describe('kuery functions', function () {
expect(result.bool).to.only.have.keys('must_not');
expect(result.bool.must_not).to.eql(ast.toElasticsearchQuery(childNode, indexPattern));
});
it('should wrap a literal argument with an "is" function targeting the default_field', function () {
const literalFoo = nodeTypes.literal.buildNode('foo');
const expectedChild = ast.toElasticsearchQuery(nodeTypes.function.buildNode('is', null, 'foo'), indexPattern);
const node = nodeTypes.function.buildNode('not', literalFoo);
const result = not.toElasticsearchQuery(node, indexPattern);
const resultChild = result.bool.must_not;
expectDeepEqual(resultChild, expectedChild);
});
});
describe('toKueryExpression', function () {
it('should serialize "not" nodes with an operator syntax', function () {
const node = nodeTypes.function.buildNode('not', childNode, 'operator');
const result = not.toKueryExpression(node);
expect(result).to.be('!"response":200');
});
it('should wrap "and" and "or" sub-queries in parenthesis', function () {
const andNode = nodeTypes.function.buildNode('and', [childNode, childNode], 'operator');
const notAndNode = nodeTypes.function.buildNode('not', andNode, 'operator');
expect(not.toKueryExpression(notAndNode)).to.be('!("response":200 and "response":200)');
const orNode = nodeTypes.function.buildNode('or', [childNode, childNode], 'operator');
const notOrNode = nodeTypes.function.buildNode('not', orNode, 'operator');
expect(not.toKueryExpression(notOrNode)).to.be('!("response":200 or "response":200)');
});
it('should not wrap "and" and "or" sub-queries that use the function syntax', function () {
const andNode = nodeTypes.function.buildNode('and', [childNode, childNode], 'function');
const notAndNode = nodeTypes.function.buildNode('not', andNode, 'operator');
expect(not.toKueryExpression(notAndNode)).to.be('!and("response":200, "response":200)');
const orNode = nodeTypes.function.buildNode('or', [childNode, childNode], 'function');
const notOrNode = nodeTypes.function.buildNode('not', orNode, 'operator');
expect(not.toKueryExpression(notOrNode)).to.be('!or("response":200, "response":200)');
});
it('should throw an error for nodes with unknown or undefined serialize styles', function () {
const node = nodeTypes.function.buildNode('not', childNode, 'notValid');
expect(not.toKueryExpression)
.withArgs(node).to.throwException(/Cannot serialize "not" function as "notValid"/);
});
});
});
});

37
src/ui/public/kuery/functions/__tests__/or.js
View file

@ -4,11 +4,10 @@ import { nodeTypes } from '../../node_types';
import * as ast from '../../ast';
import StubbedLogstashIndexPatternProvider from 'fixtures/stubbed_logstash_index_pattern';
import ngMock from 'ng_mock';
import { expectDeepEqual } from '../../../../../test_utils/expect_deep_equal';
let indexPattern;
const childNode1 = nodeTypes.function.buildNode('is', 'response', 200);
const childNode1 = nodeTypes.function.buildNode('is', 'machine.os', 'osx');
const childNode2 = nodeTypes.function.buildNode('is', 'extension', 'jpg');
describe('kuery functions', function () {
@ -22,11 +21,6 @@ describe('kuery functions', function () {
describe('buildNodeParams', function () {
it('should return "arguments" and "serializeStyle" params', function () {
const result = or.buildNodeParams([childNode1, childNode2]);
expect(result).to.only.have.keys('arguments', 'serializeStyle');
});
it('arguments should contain the unmodified child nodes', function () {
const result = or.buildNodeParams([childNode1, childNode2]);
const { arguments: [ actualChildNode1, actualChildNode2 ] } = result;
@ -34,11 +28,6 @@ describe('kuery functions', function () {
expect(actualChildNode2).to.be(childNode2);
});
it('serializeStyle should default to "operator"', function () {
const { serializeStyle } = or.buildNodeParams([childNode1, childNode2]);
expect(serializeStyle).to.be('operator');
});
});
describe('toElasticsearchQuery', function () {
@ -53,15 +42,6 @@ describe('kuery functions', function () {
);
});
it('should wrap a literal argument with an "is" function targeting the default_field', function () {
const literalFoo = nodeTypes.literal.buildNode('foo');
const expectedChild = ast.toElasticsearchQuery(nodeTypes.function.buildNode('is', null, 'foo'), indexPattern);
const node = nodeTypes.function.buildNode('or', [literalFoo]);
const result = or.toElasticsearchQuery(node, indexPattern);
const resultChild = result.bool.should[0];
expectDeepEqual(resultChild, expectedChild);
});
it('should require one of the clauses to match', function () {
const node = nodeTypes.function.buildNode('or', [childNode1, childNode2]);
const result = or.toElasticsearchQuery(node, indexPattern);
@ -70,20 +50,5 @@ describe('kuery functions', function () {
});
describe('toKueryExpression', function () {
it('should serialize "or" nodes with an operator syntax', function () {
const node = nodeTypes.function.buildNode('or', [childNode1, childNode2]);
const result = or.toKueryExpression(node);
expect(result).to.be('"response":200 or "extension":"jpg"');
});
it('should throw an error for nodes with unknown or undefined serialize styles', function () {
const node = nodeTypes.function.buildNode('or', [childNode1, childNode2], 'notValid');
expect(or.toKueryExpression)
.withArgs(node).to.throwException(/Cannot serialize "or" function as "notValid"/);
});
});
});
});

89
src/ui/public/kuery/functions/__tests__/range.js
View file

@ -1,7 +1,7 @@
import expect from 'expect.js';
import { expectDeepEqual } from '../../../../../test_utils/expect_deep_equal';
import * as range from '../range';
import { nodeTypes } from '../../node_types';
import _ from 'lodash';
import StubbedLogstashIndexPatternProvider from 'fixtures/stubbed_logstash_index_pattern';
import ngMock from 'ng_mock';
@ -18,14 +18,9 @@ describe('kuery functions', function () {
describe('buildNodeParams', function () {
it('should return "arguments" and "serializeStyle" params', function () {
const result = range.buildNodeParams('bytes', { gt: 1000, lt: 8000 });
expect(result).to.only.have.keys('arguments', 'serializeStyle');
});
it('arguments should contain the provided fieldName as a literal', function () {
const result = range.buildNodeParams('bytes', { gt: 1000, lt: 8000 });
const { arguments: [ fieldName ] } = result;
const { arguments: [fieldName] } = result;
expect(fieldName).to.have.property('type', 'literal');
expect(fieldName).to.have.property('value', 'bytes');
@ -34,7 +29,7 @@ describe('kuery functions', function () {
it('arguments should contain the provided params as named arguments', function () {
const givenParams = { gt: 1000, lt: 8000, format: 'epoch_millis' };
const result = range.buildNodeParams('bytes', givenParams);
const { arguments: [ , ...params ] } = result;
const { arguments: [, ...params] } = result;
expect(params).to.be.an('array');
expect(params).to.not.be.empty();
@ -47,64 +42,58 @@ describe('kuery functions', function () {
});
});
it('serializeStyle should default to "operator"', function () {
const result = range.buildNodeParams('bytes', { gte: 1000, lte: 8000 });
const { serializeStyle } = result;
expect(serializeStyle).to.be('operator');
});
it('serializeStyle should be "function" if either end of the range is exclusive', function () {
const result = range.buildNodeParams('bytes', { gt: 1000, lt: 8000 });
const { serializeStyle } = result;
expect(serializeStyle).to.be('function');
});
});
describe('toElasticsearchQuery', function () {
it('should return an ES range query for the node\'s field and params', function () {
const expected = {
range: {
bytes: {
gt: 1000,
lt: 8000
}
bool: {
should: [
{
range: {
bytes: {
gt: 1000,
lt: 8000
}
}
}
],
minimum_should_match: 1
}
};
const node = nodeTypes.function.buildNode('range', 'bytes', { gt: 1000, lt: 8000 });
const result = range.toElasticsearchQuery(node, indexPattern);
expect(_.isEqual(expected, result)).to.be(true);
expectDeepEqual(result, expected);
});
it('should support wildcard field names', function () {
const expected = {
bool: {
should: [
{
range: {
bytes: {
gt: 1000,
lt: 8000
}
}
}
],
minimum_should_match: 1
}
};
const node = nodeTypes.function.buildNode('range', 'byt*', { gt: 1000, lt: 8000 });
const result = range.toElasticsearchQuery(node, indexPattern);
expectDeepEqual(result, expected);
});
it('should support scripted fields', function () {
const node = nodeTypes.function.buildNode('range', 'script number', { gt: 1000, lt: 8000 });
const result = range.toElasticsearchQuery(node, indexPattern);
expect(result).to.have.key('script');
});
});
describe('toKueryExpression', function () {
it('should serialize "range" nodes with an operator syntax', function () {
const node = nodeTypes.function.buildNode('range', 'bytes', { gte: 1000, lte: 8000 }, 'operator');
const result = range.toKueryExpression(node);
expect(result).to.be('"bytes":[1000 to 8000]');
});
it('should throw an error for nodes with unknown or undefined serialize styles', function () {
const node = nodeTypes.function.buildNode('range', 'bytes', { gte: 1000, lte: 8000 }, 'notValid');
expect(range.toKueryExpression)
.withArgs(node).to.throwException(/Cannot serialize "range" function as "notValid"/);
});
it('should not support exclusive ranges in the operator syntax', function () {
const node = nodeTypes.function.buildNode('range', 'bytes', { gt: 1000, lt: 8000 });
node.serializeStyle = 'operator';
expect(range.toKueryExpression)
.withArgs(node).to.throwException(/Operator syntax only supports inclusive ranges/);
expect(result.bool.should[0]).to.have.key('script');
});
});

81
src/ui/public/kuery/functions/__tests__/utils/get_fields.js Normal file
View file

@ -0,0 +1,81 @@
import { getFields } from '../../utils/get_fields';
import expect from 'expect.js';
import StubbedLogstashIndexPatternProvider from 'fixtures/stubbed_logstash_index_pattern';
import ngMock from 'ng_mock';
import { nodeTypes } from 'ui/kuery';
let indexPattern;
describe('getFields', function () {
beforeEach(ngMock.module('kibana'));
beforeEach(ngMock.inject(function (Private) {
indexPattern = Private(StubbedLogstashIndexPatternProvider);
}));
describe('field names without a wildcard', function () {
it('should thrown an error if the field does not exist in the index pattern', function () {
const fieldNameNode = nodeTypes.literal.buildNode('nonExistentField');
expect(getFields).withArgs(fieldNameNode, indexPattern).to.throwException(
/Field nonExistentField does not exist in index pattern logstash-\*/
);
});
it('should return the single matching field in an array', function () {
const fieldNameNode = nodeTypes.literal.buildNode('extension');
const results = getFields(fieldNameNode, indexPattern);
expect(results).to.be.an('array');
expect(results).to.have.length(1);
expect(results[0].name).to.be('extension');
});
it('should not match a wildcard in a literal node', function () {
const indexPatternWithWildField = {
title: 'wildIndex',
fields: {
byName: {
'foo*': {
name: 'foo*'
}
}
}
};
const fieldNameNode = nodeTypes.literal.buildNode('foo*');
const results = getFields(fieldNameNode, indexPatternWithWildField);
expect(results).to.be.an('array');
expect(results).to.have.length(1);
expect(results[0].name).to.be('foo*');
// ensure the wildcard is not actually being parsed
expect(getFields).withArgs(nodeTypes.literal.buildNode('fo*'), indexPatternWithWildField).to.throwException(
/Field fo\* does not exist in index pattern wildIndex/
);
});
});
describe('field name patterns with a wildcard', function () {
it('should thrown an error if the pattern does not match any fields in the index pattern', function () {
const fieldNameNode = nodeTypes.wildcard.buildNode('nonExistent*');
expect(getFields).withArgs(fieldNameNode, indexPattern).to.throwException(
/No fields match the pattern nonExistent\* in index pattern logstash-\*/
);
});
it('should return all fields that match the pattern in an array', function () {
const fieldNameNode = nodeTypes.wildcard.buildNode('machine*');
const results = getFields(fieldNameNode, indexPattern);
expect(results).to.be.an('array');
expect(results).to.have.length(2);
expect(results.find((field) => {
return field.name === 'machine.os';
})).to.be.ok();
expect(results.find((field) => {
return field.name === 'machine.os.raw';
})).to.be.ok();
});
});
});

28
src/ui/public/kuery/functions/and.js
View file

@ -1,10 +1,8 @@
import * as ast from '../ast';
import { nodeTypes } from '../node_types';
export function buildNodeParams(children, serializeStyle = 'operator') {
export function buildNodeParams(children) {
return {
arguments: children,
serializeStyle
};
}
@ -14,33 +12,9 @@ export function toElasticsearchQuery(node, indexPattern) {
return {
bool: {
filter: children.map((child) => {
if (child.type === 'literal') {
child = nodeTypes.function.buildNode('is', null, child.value);
}
return ast.toElasticsearchQuery(child, indexPattern);
})
}
};
}
export function toKueryExpression(node) {
if (!['operator', 'implicit'].includes(node.serializeStyle)) {
throw new Error(`Cannot serialize "and" function as "${node.serializeStyle}"`);
}
const queryStrings = (node.arguments || []).map((arg) => {
const query = ast.toKueryExpression(arg);
if (arg.type === 'function' && arg.function === 'or') {
return `(${query})`;
}
return query;
});
if (node.serializeStyle === 'implicit') {
return queryStrings.join(' ');
}
if (node.serializeStyle === 'operator') {
return queryStrings.join(' and ');
}
}

112
src/ui/public/kuery/functions/is.js
View file

@ -1,8 +1,11 @@
import _ from 'lodash';
import * as ast from '../ast';
import * as literal from '../node_types/literal';
import * as wildcard from '../node_types/wildcard';
import { getPhraseScript } from 'ui/filter_manager/lib/phrase';
import { getFields } from './utils/get_fields';
export function buildNodeParams(fieldName, value, serializeStyle = 'operator') {
export function buildNodeParams(fieldName, value, isPhrase = false) {
if (_.isUndefined(fieldName)) {
throw new Error('fieldName is a required argument');
}
@ -10,69 +13,80 @@ export function buildNodeParams(fieldName, value, serializeStyle = 'operator') {
throw new Error('value is a required argument');
}
const fieldNode = typeof fieldName === 'string' ? ast.fromLiteralExpression(fieldName) : literal.buildNode(fieldName);
const valueNode = typeof value === 'string' ? ast.fromLiteralExpression(value) : literal.buildNode(value);
const isPhraseNode = literal.buildNode(isPhrase);
return {
arguments: [literal.buildNode(fieldName), literal.buildNode(value)],
serializeStyle
arguments: [fieldNode, valueNode, isPhraseNode],
};
}
export function toElasticsearchQuery(node, indexPattern) {
const { arguments: [ fieldNameArg, valueArg ] } = node;
const fieldName = literal.toElasticsearchQuery(fieldNameArg);
const field = indexPattern.fields.byName[fieldName];
const value = !_.isUndefined(valueArg) ? literal.toElasticsearchQuery(valueArg) : valueArg;
const { arguments: [ fieldNameArg, valueArg, isPhraseArg ] } = node;
if (field && field.scripted) {
return {
script: {
...getPhraseScript(field, value)
}
};
}
else if (fieldName === null) {
const value = !_.isUndefined(valueArg) ? ast.toElasticsearchQuery(valueArg) : valueArg;
const type = isPhraseArg.value ? 'phrase' : 'best_fields';
if (fieldNameArg.value === null) {
return {
multi_match: {
type,
query: value,
type: 'phrase',
lenient: true,
}
};
}
else if (fieldName === '*' && value === '*') {
const fields = getFields(fieldNameArg, indexPattern);
const isExistsQuery = valueArg.type === 'wildcard' && value === '*';
const isMatchAllQuery = isExistsQuery && fields && fields.length === indexPattern.fields.length;
if (isMatchAllQuery) {
return { match_all: {} };
}
else if (fieldName === '*' && value !== '*') {
return {
multi_match: {
query: value,
fields: ['*'],
type: 'phrase',
lenient: true,
const queries = fields.reduce((accumulator, field) => {
if (field.scripted) {
// Exists queries don't make sense for scripted fields
if (!isExistsQuery) {
return [...accumulator, {
script: {
...getPhraseScript(field, value)
}
}];
}
};
}
else if (fieldName !== '*' && value === '*') {
return {
exists: { field: fieldName }
};
}
else {
return {
match_phrase: {
[fieldName]: value
}
};
}
}
else if (isExistsQuery) {
return [...accumulator, {
exists: {
field: field.name
}
}];
}
else if (valueArg.type === 'wildcard') {
return [...accumulator, {
query_string: {
fields: [field.name],
query: wildcard.toQueryStringQuery(valueArg),
}
}];
}
else {
const queryType = type === 'phrase' ? 'match_phrase' : 'match';
return [...accumulator, {
[queryType]: {
[field.name]: value
}
}];
}
}, []);
return {
bool: {
should: queries,
minimum_should_match: 1
}
};
}
export function toKueryExpression(node) {
if (node.serializeStyle !== 'operator') {
throw new Error(`Cannot serialize "is" function as "${node.serializeStyle}"`);
}
const { arguments: [ fieldNameArg, valueArg ] } = node;
const fieldName = literal.toKueryExpression(fieldNameArg);
const value = !_.isUndefined(valueArg) ? literal.toKueryExpression(valueArg) : valueArg;
return `${fieldName}:${value}`;
}

28
src/ui/public/kuery/functions/not.js
View file

@ -1,18 +1,13 @@
import * as ast from '../ast';
import { nodeTypes } from '../node_types';
export function buildNodeParams(child, serializeStyle = 'operator') {
export function buildNodeParams(child) {
return {
arguments: [child],
serializeStyle
};
}
export function toElasticsearchQuery(node, indexPattern) {
let [ argument ] = node.arguments;
if (argument.type === 'literal') {
argument = nodeTypes.function.buildNode('is', null, argument.value);
}
const [ argument ] = node.arguments;
return {
bool: {
@ -21,22 +16,3 @@ export function toElasticsearchQuery(node, indexPattern) {
};
}
export function toKueryExpression(node) {
if (node.serializeStyle !== 'operator') {
throw new Error(`Cannot serialize "not" function as "${node.serializeStyle}"`);
}
const [ argument ] = node.arguments;
const queryString = ast.toKueryExpression(argument);
if (
argument.function &&
(argument.function === 'and' || argument.function === 'or') &&
argument.serializeStyle !== 'function'
) {
return `!(${queryString})`;
}
else {
return `!${queryString}`;
}
}

21
src/ui/public/kuery/functions/or.js
View file

@ -1,10 +1,8 @@
import * as ast from '../ast';
import { nodeTypes } from '../node_types';
export function buildNodeParams(children, serializeStyle = 'operator') {
export function buildNodeParams(children) {
return {
arguments: children,
serializeStyle,
};
}
@ -14,26 +12,9 @@ export function toElasticsearchQuery(node, indexPattern) {
return {
bool: {
should: children.map((child) => {
if (child.type === 'literal') {
child = nodeTypes.function.buildNode('is', null, child.value);
}
return ast.toElasticsearchQuery(child, indexPattern);
}),
minimum_should_match: 1,
},
};
}
export function toKueryExpression(node) {
if (node.serializeStyle !== 'operator') {
throw new Error(`Cannot serialize "or" function as "${node.serializeStyle}"`);
}
const queryStrings = (node.arguments || []).map((arg) => {
return ast.toKueryExpression(arg);
});
return queryStrings.join(' or ');
}

50
src/ui/public/kuery/functions/range.js
View file

@ -2,62 +2,50 @@ import _ from 'lodash';
import { nodeTypes } from '../node_types';
import * as ast from '../ast';
import { getRangeScript } from 'ui/filter_manager/lib/range';
import { getFields } from './utils/get_fields';
export function buildNodeParams(fieldName, params, serializeStyle = 'operator') {
export function buildNodeParams(fieldName, params) {
params = _.pick(params, 'gt', 'lt', 'gte', 'lte', 'format');
const fieldNameArg = nodeTypes.literal.buildNode(fieldName);
const fieldNameArg = typeof fieldName === 'string' ? ast.fromLiteralExpression(fieldName) : nodeTypes.literal.buildNode(fieldName);
const args = _.map(params, (value, key) => {
return nodeTypes.namedArg.buildNode(key, value);
});
// we only support inclusive ranges in the operator syntax currently
if (_.has(params, 'gt') || _.has(params, 'lt')) {
serializeStyle = 'function';
}
return {
arguments: [fieldNameArg, ...args],
serializeStyle,
};
}
export function toElasticsearchQuery(node, indexPattern) {
const [ fieldNameArg, ...args ] = node.arguments;
const fieldName = nodeTypes.literal.toElasticsearchQuery(fieldNameArg);
const field = indexPattern.fields.byName[fieldName];
const fields = getFields(fieldNameArg, indexPattern);
const namedArgs = extractArguments(args);
const queryParams = _.mapValues(namedArgs, ast.toElasticsearchQuery);
if (field && field.scripted) {
const queries = fields.map((field) => {
if (field.scripted) {
return {
script: {
...getRangeScript(field, queryParams)
}
};
}
return {
script: {
...getRangeScript(field, queryParams)
range: {
[field.name]: queryParams
}
};
}
});
return {
range: {
[fieldName]: queryParams
bool: {
should: queries,
minimum_should_match: 1
}
};
}
export function toKueryExpression(node) {
if (node.serializeStyle !== 'operator') {
throw new Error(`Cannot serialize "range" function as "${node.serializeStyle}"`);
}
const [ fieldNameArg, ...args ] = node.arguments;
const fieldName = ast.toKueryExpression(fieldNameArg);
const { gte, lte } = extractArguments(args);
if (_.isUndefined(gte) || _.isUndefined(lte)) {
throw new Error(`Operator syntax only supports inclusive ranges`);
}
return `${fieldName}:[${ast.toKueryExpression(gte)} to ${ast.toKueryExpression(lte)}]`;
}
function extractArguments(args) {
if ((args.gt && args.gte) || (args.lt && args.lte)) {
throw new Error('range ends cannot be both inclusive and exclusive');

19
src/ui/public/kuery/functions/utils/get_fields.js Normal file
View file

@ -0,0 +1,19 @@
import * as literal from '../../node_types/literal';
import * as wildcard from '../../node_types/wildcard';
export function getFields(node, indexPattern) {
if (node.type === 'literal') {
const fieldName = literal.toElasticsearchQuery(node);
const field = indexPattern.fields.byName[fieldName];
if (!field) {
throw new Error(`Field ${fieldName} does not exist in index pattern ${indexPattern.title}`);
}
return [field];
} else if (node.type === 'wildcard') {
const fields = indexPattern.fields.filter(field => wildcard.test(node, field.name));
if (fields.length === 0) {
throw new Error(`No fields match the pattern ${wildcard.toElasticsearchQuery(node)} in index pattern ${indexPattern.title}`);
}
return fields;
}
}

34
src/ui/public/kuery/node_types/__tests__/function.js
View file

@ -21,7 +21,7 @@ describe('kuery node types', function () {
describe('buildNode', function () {
it('should return a node representing the given kuery function', function () {
const result = functionType.buildNode('is', 'response', 200);
const result = functionType.buildNode('is', 'extension', 'jpg');
expect(result).to.have.property('type', 'function');
expect(result).to.have.property('function', 'is');
expect(result).to.have.property('arguments');
@ -32,8 +32,8 @@ describe('kuery node types', function () {
describe('buildNodeWithArgumentNodes', function () {
it('should return a function node with the given argument list untouched', function () {
const fieldNameLiteral = nodeTypes.literal.buildNode('response');
const valueLiteral = nodeTypes.literal.buildNode(200);
const fieldNameLiteral = nodeTypes.literal.buildNode('extension');
const valueLiteral = nodeTypes.literal.buildNode('jpg');
const argumentNodes = [fieldNameLiteral, valueLiteral];
const result = functionType.buildNodeWithArgumentNodes('is', argumentNodes);
@ -49,7 +49,7 @@ describe('kuery node types', function () {
describe('toElasticsearchQuery', function () {
it('should return the given function type\'s ES query representation', function () {
const node = functionType.buildNode('is', 'response', 200);
const node = functionType.buildNode('is', 'extension', 'jpg');
const expected = isFunction.toElasticsearchQuery(node, indexPattern);
const result = functionType.toElasticsearchQuery(node, indexPattern);
expect(_.isEqual(expected, result)).to.be(true);
@ -57,32 +57,6 @@ describe('kuery node types', function () {
});
describe('toKueryExpression', function () {
it('should return the function syntax representation of the given node by default', function () {
const node = functionType.buildNode('exists', 'foo');
expect(functionType.toKueryExpression(node)).to.be('exists("foo")');
});
it('should return the function syntax representation of the given node if serializeStyle is "function"', function () {
const node = functionType.buildNode('exists', 'foo');
node.serializeStyle = 'function';
expect(functionType.toKueryExpression(node)).to.be('exists("foo")');
});
it('should defer to the function\'s serializer if another serializeStyle is specified', function () {
const node = functionType.buildNode('is', 'response', 200);
expect(node.serializeStyle).to.be('operator');
expect(functionType.toKueryExpression(node)).to.be('"response":200');
});
it('should simply return the node\'s "text" property if one exists', function () {
const node = functionType.buildNode('exists', 'foo');
node.text = 'bar';
expect(functionType.toKueryExpression(node)).to.be('bar');
});
});
});

13
src/ui/public/kuery/node_types/__tests__/literal.js
View file

@ -25,19 +25,6 @@ describe('kuery node types', function () {
});
describe('toKueryExpression', function () {
it('should return the literal value represented by the given node', function () {
const numberNode = literal.buildNode(200);
expect(literal.toKueryExpression(numberNode)).to.be(200);
});
it('should wrap string values in double quotes', function () {
const stringNode = literal.buildNode('foo');
expect(literal.toKueryExpression(stringNode)).to.be('"foo"');
});
});
});

9
src/ui/public/kuery/node_types/__tests__/named_arg.js
View file

@ -39,15 +39,6 @@ describe('kuery node types', function () {
});
describe('toKueryExpression', function () {
it('should return the argument name and value represented by the given node', function () {
const node = namedArg.buildNode('fieldName', 'foo');
expect(namedArg.toKueryExpression(node)).to.be('fieldName="foo"');
});
});
});
});

78
src/ui/public/kuery/node_types/__tests__/wildcard.js Normal file
View file

@ -0,0 +1,78 @@
import expect from 'expect.js';
import * as wildcard from '../wildcard';
describe('kuery node types', function () {
describe('wildcard', function () {
describe('buildNode', function () {
it('should accept an array argument representing a wildcard string', function () {
const wildcardValue = [
'foo',
Symbol('*'),
'bar',
];
const result = wildcard.buildNode(wildcardValue);
expect(result).to.have.property('type', 'wildcard');
expect(result).to.have.property('value', wildcardValue);
});
it('should accept and parse a wildcard string', function () {
const result = wildcard.buildNode('foo*bar');
expect(result).to.have.property('type', 'wildcard');
expect(result.value[0]).to.be('foo');
expect(result.value[1]).to.be.a('symbol');
expect(result.value[1].toString()).to.be('Symbol(*)');
expect(result.value[2]).to.be('bar');
});
});
describe('toElasticsearchQuery', function () {
it('should return the string representation of the wildcard literal', function () {
const node = wildcard.buildNode('foo*bar');
const result = wildcard.toElasticsearchQuery(node);
expect(result).to.be('foo*bar');
});
});
describe('toQueryStringQuery', function () {
it('should return the string representation of the wildcard literal', function () {
const node = wildcard.buildNode('foo*bar');
const result = wildcard.toQueryStringQuery(node);
expect(result).to.be('foo*bar');
});
it('should escape query_string query special characters other than wildcard', function () {
const node = wildcard.buildNode('+foo*bar');
const result = wildcard.toQueryStringQuery(node);
expect(result).to.be('\\+foo*bar');
});
});
describe('test', function () {
it('should return a boolean indicating whether the string matches the given wildcard node', function () {
const node = wildcard.buildNode('foo*bar');
expect(wildcard.test(node, 'foobar')).to.be(true);
expect(wildcard.test(node, 'foobazbar')).to.be(true);
expect(wildcard.test(node, 'foobar')).to.be(true);
expect(wildcard.test(node, 'fooqux')).to.be(false);
expect(wildcard.test(node, 'bazbar')).to.be(false);
});
});
});
});

21
src/ui/public/kuery/node_types/function.js
View file

@ -1,6 +1,5 @@
import _ from 'lodash';
import { functions } from '../functions';
import { nodeTypes } from '../node_types';
export function buildNode(functionName, ...functionArgs) {
const kueryFunction = functions[functionName];
@ -17,7 +16,7 @@ export function buildNode(functionName, ...functionArgs) {
}
// Mainly only useful in the grammar where we'll already have real argument nodes in hand
export function buildNodeWithArgumentNodes(functionName, argumentNodes, serializeStyle = 'function') {
export function buildNodeWithArgumentNodes(functionName, argumentNodes) {
if (_.isUndefined(functions[functionName])) {
throw new Error(`Unknown function "${functionName}"`);
}
@ -26,7 +25,6 @@ export function buildNodeWithArgumentNodes(functionName, argumentNodes, serializ
type: 'function',
function: functionName,
arguments: argumentNodes,
serializeStyle
};
}
@ -35,20 +33,3 @@ export function toElasticsearchQuery(node, indexPattern) {
return kueryFunction.toElasticsearchQuery(node, indexPattern);
}
export function toKueryExpression(node) {
const kueryFunction = functions[node.function];
if (!_.isUndefined(node.text)) {
return node.text;
}
if (node.serializeStyle && node.serializeStyle !== 'function') {
return kueryFunction.toKueryExpression(node);
}
const functionArguments = (node.arguments || []).map((argument) => {
return nodeTypes[argument.type].toKueryExpression(argument);
});
return `${node.function}(${functionArguments.join(', ')})`;
}

3
src/ui/public/kuery/node_types/index.js
View file

@ -1,10 +1,11 @@
import * as functionType from './function';
import * as literal from './literal';
import * as namedArg from './named_arg';
import * as wildcard from './wildcard';
export const nodeTypes = {
function: functionType,
literal,
namedArg,
wildcard,
};

10
src/ui/public/kuery/node_types/literal.js
View file

@ -1,5 +1,3 @@
import _ from 'lodash';
export function buildNode(value) {
return {
type: 'literal',
@ -11,11 +9,3 @@ export function toElasticsearchQuery(node) {
return node.value;
}
export function toKueryExpression(node) {
if (_.isString(node.value)) {
const escapedValue = node.value.replace(/"/g, '\\"');
return `"${escapedValue}"`;
}
return node.value;
}

3
src/ui/public/kuery/node_types/named_arg.js
View file

@ -15,6 +15,3 @@ export function toElasticsearchQuery(node) {
return ast.toElasticsearchQuery(node.value);
}
export function toKueryExpression(node) {
return `${node.name}=${ast.toKueryExpression(node.value)}`;
}

59
src/ui/public/kuery/node_types/wildcard.js Normal file
View file

@ -0,0 +1,59 @@
import { fromLiteralExpression } from '../ast/ast';
export const wildcardSymbol = Symbol('*');
// Copied from https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions
function escapeRegExp(string) {
return string.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); // $& means the whole matched string
}
// See https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#_reserved_characters
function escapeQueryString(string) {
return string.replace(/[+-=&|><!(){}[\]^"~*?:\\/]/g, '\\$&'); // $& means the whole matched string
}
export function buildNode(value) {
if (typeof value === 'string') {
return fromLiteralExpression(value);
}
return {
type: 'wildcard',
value,
};
}
export function test(node, string) {
const { value } = node;
const regex = value.map(sequence => {
if (typeof sequence === 'symbol') {
return '.*';
} else {
return escapeRegExp(sequence);
}
}).join('');
const regexp = new RegExp(`^${regex}$`);
return regexp.test(string);
}
export function toElasticsearchQuery(node) {
const { value } = node;
return value.map(sequence => {
if (typeof sequence === 'symbol') {
return '*';
} else {
return sequence;
}
}).join('');
}
export function toQueryStringQuery(node) {
const { value } = node;
return value.map(sequence => {
if (typeof sequence === 'symbol') {
return '*';
} else {
return escapeQueryString(sequence);
}
}).join('');
}

16
src/ui/public/query_bar/directive/query_bar.html
View file

@ -79,22 +79,6 @@
</div>
</div>
<!-- kql input -->
<div class="kuiLocalSearchAssistedInput" ng-if="queryBar.localQuery.language === 'kql'">
<input
ng-model="queryBar.localQuery.query"
input-focus
disable-input-focus="queryBar.disableAutoFocus"
kbn-typeahead-input
placeholder="Search... (e.g. status:200 AND extension:PHP)"
aria-label="Search input"
aria-describedby="discoverKuerySyntaxHint"
type="text"
class="kuiLocalSearchInput"
ng-class="{'kuiLocalSearchInput-isInvalid': queryBarForm.$invalid}"
data-test-subj="queryInput"
/>
</div>
<select
class="kuiLocalSearchSelect"

1
src/ui/public/query_bar/lib/queryLanguages.js
View file

@ -1,5 +1,4 @@
export const queryLanguages = [
'lucene',
'kuery',
// 'kql'
];

6
test/functional/apps/discover/_discover.js
View file

@ -281,6 +281,8 @@ export default function ({ getService, getPageObjects }) {
await PageObjects.discover.clickFieldListItem('response');
await PageObjects.discover.clickFieldListPlusFilter('response', 200);
await PageObjects.header.waitUntilLoadingHasFinished();
await PageObjects.discover.query('php');
await PageObjects.header.waitUntilLoadingHasFinished();
let queryString = await queryBar.getQueryString();
expect(queryString).to.not.be.empty();
@ -293,8 +295,10 @@ export default function ({ getService, getPageObjects }) {
await PageObjects.discover.clickFieldListPlusFilter('response', 200);
await PageObjects.header.waitUntilLoadingHasFinished();
await PageObjects.discover.query('php');
queryString = await queryBar.getQueryString();
expect(queryString).to.be.empty();
await PageObjects.header.waitUntilLoadingHasFinished();
expect(queryString).to.not.be.empty();
expect(await filterBar.hasFilter('response', 200)).to.be(true);
await queryBar.setLanguage('kuery');