github-mirrors/kibana
1
0
Fork 0
mirror of https://github.com/elastic/kibana.git synced 2025-04-24 01:38:56 -04:00
Code Issues Projects Releases Packages Wiki Activity

Exclusion of source fields (#8920)

Browse source 
Backports PR #7402

**Commit 1:**
added source filtering

* Original sha: e7204bef63
* Authored by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-01-12T10:08:35Z

**Commit 2:**
ditched the new 'retrieved fields' tab and added checkbox to exclude a field in the field control

* Original sha: b228c66c01
* Authored by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-01-13T14:44:38Z

**Commit 3:**
disable field exclusion checkbox if field is a metafield

* Original sha: 0d062209fb
* Authored by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-01-15T15:39:47Z

**Commit 4:**
[indexPattern] copy excluded field property when refreshing fields

* Original sha: c3c1d4823b
* Authored by spalger <spalger@users.noreply.github.com> on 2016-01-21T16:01:48Z
* Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T20:40:15Z

**Commit 5:**
[indexPattern/field] move isMetaField consideration into Field

* Original sha: 5b0b00adfb
* Authored by spalger <spalger@users.noreply.github.com> on 2016-01-21T16:22:52Z
* Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T20:44:37Z

**Commit 6:**
[indexPattern/edit] invert the "retreived" column, for accuracy

* Original sha: 1dee6f173f
* Authored by spalger <spalger@users.noreply.github.com> on 2016-01-21T16:24:44Z
* Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T20:47:21Z

**Commit 7:**
[indexPattern/field] touchup the field.exclude message

* Original sha: ec0c3341d1
* Authored by spalger <spalger@users.noreply.github.com> on 2016-01-21T16:25:24Z
* Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T20:47:23Z

**Commit 8:**
Fix typo

* Original sha: 062931a3f4
* Authored by Spencer <spalger@users.noreply.github.com> on 2016-01-21T19:33:14Z
* Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T20:48:53Z

**Commit 9:**
[indexPattern] handle index patterns without fields

* Original sha: 9d70208f56
* Authored by spalger <spalger@users.noreply.github.com> on 2016-01-22T18:27:45Z
* Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T20:51:11Z

**Commit 10:**
[courier/searchSource] auto add source filter for index pattern

* Original sha: 5a97b6b668
* Authored by spalger <spalger@users.noreply.github.com> on 2016-01-23T01:08:52Z
* Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T21:04:09Z

**Commit 11:**
[docTable] remove irrelevant test about source filtering

* Original sha: 8211990a01
* Authored by spalger <spalger@users.noreply.github.com> on 2016-02-12T08:17:56Z
* Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T21:04:11Z

**Commit 12:**
[settings/indices] cleanup imports

* Original sha: fa5b22d9ab
* Authored by spalger <spalger@users.noreply.github.com> on 2016-04-05T00:29:36Z
* Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T21:06:11Z

**Commit 13:**
[settings/indexPattern/fields] add "field filters" tab

* Original sha: 7819fa984d
* Authored by spalger <spalger@users.noreply.github.com> on 2016-04-05T06:07:31Z
* Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T21:11:20Z

**Commit 14:**
[imports] fix old testUtils import

* Original sha: d4b9c116a6
* Authored by spalger <spalger@users.noreply.github.com> on 2016-04-05T06:08:46Z
* Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T21:11:24Z

**Commit 15:**
[ui/fieldWildcard] add lib to match names based on field-style wildcards

* Original sha: d75a63fa34
* Authored by spalger <spalger@users.noreply.github.com> on 2016-04-05T08:02:01Z
* Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T21:11:24Z

**Commit 16:**
[settings/fieldFilters] list filter matches, remove excluded fields from fieldata_fields

* Original sha: 08f8fd7e0d
* Authored by spalger <spalger@users.noreply.github.com> on 2016-04-05T08:59:39Z
* Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T21:11:24Z

**Commit 17:**
[fieldWildcard] properly escape regexp control chars

* Original sha: 385a0620c0
* Authored by spalger <spalger@users.noreply.github.com> on 2016-04-05T09:13:21Z
* Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T21:11:25Z

**Commit 18:**
[settings/indexPatterns] mark fields excluded if they match an exclude pattern

* Original sha: da62b2a761
* Authored by spalger <spalger@users.noreply.github.com> on 2016-04-05T09:29:03Z
* Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T21:11:25Z

**Commit 19:**
[fieldWildcard] properly bind the regexp to the ends

* Original sha: e89ceeabdc
* Authored by spalger <spalger@users.noreply.github.com> on 2016-04-05T23:59:06Z
* Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T21:11:25Z

**Commit 20:**
[indexPattern] remove unneeded lodash chain

* Original sha: 14d5458bb7
* Authored by spalger <spalger@users.noreply.github.com> on 2016-04-06T00:00:28Z
* Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T21:11:25Z

**Commit 21:**
[settings/indices] use settings-indices- prefix for tab direcives

* Original sha: dc53a103bc
* Authored by spalger <spalger@users.noreply.github.com> on 2016-04-06T00:03:45Z
* Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T21:12:19Z

**Commit 22:**
corrected rebase on master

* Original sha: 4be31b2878
* Authored by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T21:29:19Z

**Commit 23:**
Do not match exclusion on meta/scripted fields. Disable filter bar when on 'Filter fields' tab. Removed exclusion checkbox in the field controls page. Corrected typos. Improved documentation phrasing.

* Original sha: 79e87ca3ae
* Authored by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-09T08:14:12Z

**Commit 24:**
Merge branch 'master' into issue4366

* Original sha: 984d3349ba
* Authored by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-09T08:29:38Z

**Commit 25:**
corrected error in merge with _index_pattern

* Original sha: 3375fd1f02
* Authored by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-09T08:35:25Z

**Commit 26:**
removed unused code

* Original sha: 6a0b331297
* Authored by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-09T08:56:56Z

**Commit 27:**
Merge branch 'master' into issue4366

* Original sha: 81a56f9f05
* Authored by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-09T21:24:12Z

**Commit 28:**
added missing fieldFilters to test dumps

* Original sha: c209ce3165
* Authored by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-11T09:14:10Z

**Commit 29:**
Merge branch 'master' into issue4366

* Original sha: 95114865b9
* Authored by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-13T08:57:50Z

**Commit 30:**
added the management section refactoring from master

* Original sha: 9e32bbc2cf
* Authored by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-30T09:14:19Z
This commit is contained in:
jasper 2016-11-01 13:56:28 -04:00 • committed by Spencer
parent 9eb4f5bbba
commit b3e0138d66
32 changed files with 639 additions and 54 deletions
Download patch file Download diff file Expand all files Collapse all files

2
src/core_plugins/kibana/public/doc/controllers/doc.js
View file

@ -51,7 +51,7 @@ app.controller('doc', function ($scope, $route, es, timefilter) {
}
},
stored_fields: computedFields.storedFields,
_source: computedFields._source,
_source: true,
script_fields: computedFields.scriptFields,
docvalue_fields: computedFields.docvalueFields
}

1
src/core_plugins/kibana/public/management/index.js
View file

@ -6,6 +6,7 @@ import 'ui/filters/start_from';
import 'ui/field_editor';
import 'plugins/kibana/management/sections/indices/_indexed_fields';
import 'plugins/kibana/management/sections/indices/_scripted_fields';
import 'plugins/kibana/management/sections/indices/source_filters/source_filters';
import 'ui/directives/bread_crumbs';
import uiRoutes from 'ui/routes';
import uiModules from 'ui/modules';

25
src/core_plugins/kibana/public/management/sections/indices/_edit.html
View file

@ -37,17 +37,30 @@
<br />
<!-- tab list -->
<ul class="nav nav-tabs">
<li class="kbn-management-tab" ng-class="{ active: state.tab === fieldType.index }" ng-repeat="fieldType in fieldTypes">
<a ng-click="changeTab(fieldType)">
{{ fieldType.title }}
<small>({{ fieldType.count }})</small>
<li class="kbn-management-tab" ng-class="{ active: state.tab === editSection.index }" ng-repeat="editSection in editSections">
<a ng-click="changeTab(editSection)">
{{ editSection.title }}
<small>({{ editSection.count }})</small>
</a>
</li>
</ul>
<indexed-fields ng-show="state.tab == 'indexedFields'" class="fields indexed-fields"></indexed-fields>
<scripted-fields ng-show="state.tab == 'scriptedFields'" class="fields scripted-fields"></scripted-fields>
<!-- tabs -->
<indexed-fields
ng-show="state.tab == 'indexedFields'"
class="fields indexed-fields"
></indexed-fields>
<scripted-fields
ng-show="state.tab == 'scriptedFields'"
class="fields scripted-fields"
></scripted-fields>
<source-filters
ng-show="state.tab == 'sourceFilters'"
index-pattern="indexPattern"
class="fields source-filters"
></source-filters>
</div>
</kbn-management-indices>

8
src/core_plugins/kibana/public/management/sections/indices/_edit.js
View file

@ -1,10 +1,11 @@
import _ from 'lodash';
import 'plugins/kibana/management/sections/indices/_indexed_fields';
import 'plugins/kibana/management/sections/indices/_scripted_fields';
import 'plugins/kibana/management/sections/indices/source_filters/source_filters';
import 'plugins/kibana/management/sections/indices/_index_header';
import RefreshKibanaIndex from 'plugins/kibana/management/sections/indices/_refresh_kibana_index';
import UrlProvider from 'ui/url';
import IndicesFieldTypesProvider from 'plugins/kibana/management/sections/indices/_field_types';
import IndicesEditSectionsProvider from 'plugins/kibana/management/sections/indices/_edit_sections';
import uiRoutes from 'ui/routes';
import uiModules from 'ui/modules';
import editTemplate from 'plugins/kibana/management/sections/indices/_edit.html';
@ -51,9 +52,8 @@ uiModules.get('apps/management')
docTitle.change($scope.indexPattern.id);
const otherIds = _.without($route.current.locals.indexPatternIds, $scope.indexPattern.id);
const fieldTypes = Private(IndicesFieldTypesProvider);
$scope.$watch('indexPattern.fields', function () {
$scope.fieldTypes = fieldTypes($scope.indexPattern);
$scope.editSections = Private(IndicesEditSectionsProvider)($scope.indexPattern);
});
$scope.changeTab = function (obj) {
@ -62,7 +62,7 @@ uiModules.get('apps/management')
};
$scope.$watch('state.tab', function (tab) {
if (!tab) $scope.changeTab($scope.fieldTypes[0]);
if (!tab) $scope.changeTab($scope.editSections[0]);
});
$scope.$watchCollection('indexPattern.fields', function () {

33
src/core_plugins/kibana/public/management/sections/indices/_edit_sections.js Normal file
View file

@ -0,0 +1,33 @@
import _ from 'lodash';
export default function GetFieldTypes() {
return function (indexPattern) {
const fieldCount = _.countBy(indexPattern.fields, function (field) {
return (field.scripted) ? 'scripted' : 'indexed';
});
_.defaults(fieldCount, {
indexed: 0,
scripted: 0,
sourceFilters: 0
});
return [
{
title: 'fields',
index: 'indexedFields',
count: fieldCount.indexed
},
{
title: 'scripted fields',
index: 'scriptedFields',
count: fieldCount.scripted
},
{
title: 'source filters',
index: 'sourceFilters',
count: fieldCount.sourceFilters
}
];
};
};

24
src/core_plugins/kibana/public/management/sections/indices/_field_types.js
View file

@ -1,24 +0,0 @@
import _ from 'lodash';
export default function GetFieldTypes() {
return function (indexPattern) {
const fieldCount = _.countBy(indexPattern.fields, function (field) {
return (field.scripted) ? 'scripted' : 'indexed';
});
_.defaults(fieldCount, {
indexed: 0,
scripted: 0
});
return [{
title: 'fields',
index: 'indexedFields',
count: fieldCount.indexed
}, {
title: 'scripted fields',
index: 'scriptedFields',
count: fieldCount.scripted
}];
};
};

16
src/core_plugins/kibana/public/management/sections/indices/_indexed_fields.js
View file

@ -4,13 +4,15 @@ import nameHtml from 'plugins/kibana/management/sections/indices/_field_name.htm
import typeHtml from 'plugins/kibana/management/sections/indices/_field_type.html';
import controlsHtml from 'plugins/kibana/management/sections/indices/_field_controls.html';
import uiModules from 'ui/modules';
import FieldWildcardProvider from 'ui/field_wildcard';
import indexedFieldsTemplate from 'plugins/kibana/management/sections/indices/_indexed_fields.html';
uiModules.get('apps/management')
.directive('indexedFields', function ($filter) {
.directive('indexedFields', function (Private, $filter) {
const yesTemplate = '<i class="fa fa-check" aria-label="yes"></i>';
const noTemplate = '';
const filter = $filter('filter');
const { fieldWildcardMatcher } = Private(FieldWildcardProvider);
return {
restrict: 'E',
@ -26,6 +28,7 @@ uiModules.get('apps/management')
{ title: 'searchable', info: 'These fields can be used in the filter bar' },
{ title: 'aggregatable' , info: 'These fields can be used in visualization aggregations' },
{ title: 'analyzed', info: 'Analyzed fields may require extra memory to visualize' },
{ title: 'excluded', info: 'Fields that are excluded from _source when it is fetched' },
{ title: 'controls', sortable: false }
];
@ -34,14 +37,17 @@ uiModules.get('apps/management')
function refreshRows() {
// clear and destroy row scopes
_.invoke(rowScopes.splice(0), '$destroy');
const fields = filter($scope.indexPattern.getNonScriptedFields(), $scope.fieldFilter);
_.find($scope.fieldTypes, {index: 'indexedFields'}).count = fields.length; // Update the tab count
const sourceFilters = $scope.indexPattern.sourceFilters && $scope.indexPattern.sourceFilters.map(f => f.value) || [];
const fieldWildcardMatch = fieldWildcardMatcher(sourceFilters);
_.find($scope.editSections, {index: 'indexedFields'}).count = fields.length; // Update the tab count
$scope.rows = fields.map(function (field) {
const childScope = _.assign($scope.$new(), { field: field });
rowScopes.push(childScope);
const excluded = fieldWildcardMatch(field.name);
return [
{
markup: nameHtml,
@ -66,6 +72,10 @@ uiModules.get('apps/management')
markup: field.analyzed ? yesTemplate : noTemplate,
value: field.analyzed
},
{
markup: excluded ? yesTemplate : noTemplate,
value: excluded
},
{
markup: controlsHtml,
scope: childScope

2
src/core_plugins/kibana/public/management/sections/indices/_scripted_fields.js
View file

@ -38,7 +38,7 @@ uiModules.get('apps/management')
rowScopes.length = 0;
const fields = filter($scope.indexPattern.getScriptedFields(), $scope.fieldFilter);
_.find($scope.fieldTypes, {index: 'scriptedFields'}).count = fields.length; // Update the tab count
_.find($scope.editSections, {index: 'scriptedFields'}).count = fields.length; // Update the tab count
$scope.rows = fields.map(function (field) {
const rowScope = $scope.$new();

26
src/core_plugins/kibana/public/management/sections/indices/source_filters/controls.html Normal file
View file

@ -0,0 +1,26 @@
<div class="actions">
<button
aria-label="Edit source filter"
ng-if="sourceFilters.editing !== filter"
ng-click="sourceFilters.editing = filter"
type="button"
class="btn btn-xs btn-default">
<i aria-hidden="true" class="fa fa-pencil"></i>
</button>
<button
aria-label="Save source filter"
ng-if="sourceFilters.editing === filter"
ng-click="sourceFilters.save()"
ng-disabled="!filter.value"
type="button"
class="btn btn-xs btn-primary">
<i aria-hidden="true" class="fa fa-save"></i>
</button>
<button
aria-label="Delete source filter"
ng-click="sourceFilters.delete(filter)"
type="button"
class="btn btn-xs btn-danger">
<i aria-hidden="true" class="fa fa-trash"></i>
</button>
</div>

12
src/core_plugins/kibana/public/management/sections/indices/source_filters/filter.html Normal file
View file

@ -0,0 +1,12 @@
<div class="value">
<span ng-if="sourceFilters.editing !== filter">{{ filter.value }}</span>
<input
ng-model="filter.value"
input-focus
ng-if="sourceFilters.editing === filter"
placeholder="{{ sourceFilters.placeHolder }}"
type="text"
required
class="form-control">
</div>

37
src/core_plugins/kibana/public/management/sections/indices/source_filters/source_filters.html Normal file
View file

@ -0,0 +1,37 @@
<h3>Source Filters</h3>
<p>
Source filters can be used to exclude one or more fields when fetching the document source. This happens when viewing a document in the Discover app, or with a table displaying results from a saved search in the Dashboard app. Each row is built using the source of a single document, and if you have documents with large or unimportant fields you may benefit from filtering those out at this lower level.
</p>
<p>
Note that multi-fields will incorrectly appear as matches in the table below. These filters only actually apply to fields in the original source document, so matching multi-fields are not actually being filtered.
</p>
<div ng-class="{ saving: sourceFilters.saving }" class="source-filters-container">
<form name="form" ng-submit="sourceFilters.create()">
<div class="input-group">
<input
ng-model="sourceFilters.newValue"
placeholder="{{ sourceFilters.placeHolder }}"
type="text"
class="form-control">
<div class="input-group-btn" role="group" aria-label="Source Filter Editor Controls">
<button
type="submit"
class="btn btn-primary"
ng-disabled="!sourceFilters.newValue">
Add
</button>
</div>
</div>
</form>
<paginated-table
columns="columns"
rows="rows"
per-page="perPage">
</paginated-table>
</div>

119
src/core_plugins/kibana/public/management/sections/indices/source_filters/source_filters.js Normal file
View file

@ -0,0 +1,119 @@
import { find, each, escape, invoke, size, without } from 'lodash';
import uiModules from 'ui/modules';
import Notifier from 'ui/notify/notifier';
import FieldWildcardProvider from 'ui/field_wildcard';
import controlsHtml from 'plugins/kibana/management/sections/indices/source_filters/controls.html';
import filterHtml from 'plugins/kibana/management/sections/indices/source_filters/filter.html';
import template from './source_filters.html';
import './source_filters.less';
const notify = new Notifier();
uiModules.get('kibana')
.directive('sourceFilters', function (Private, $filter) {
const angularFilter = $filter('filter');
const { fieldWildcardMatcher } = Private(FieldWildcardProvider);
const rowScopes = []; // track row scopes, so they can be destroyed as needed
return {
restrict: 'E',
scope: {
indexPattern: '='
},
template,
controllerAs: 'sourceFilters',
controller: class FieldFiltersController {
constructor($scope) {
if (!$scope.indexPattern) {
throw new Error('index pattern is required');
}
$scope.perPage = 25;
$scope.columns = [
{
title: 'filter'
},
{
title: 'matches',
sortable: false,
info: 'The source fields that match the filter.'
},
{
title: 'controls',
sortable: false
}
];
this.$scope = $scope;
this.saving = false;
this.editing = null;
this.newValue = null;
this.placeHolder = 'source filter, accepts wildcards (e.g., `user*` to filter fields starting with \'user\')';
$scope.$watchMulti([ '[]indexPattern.sourceFilters', '$parent.fieldFilter' ], () => {
invoke(rowScopes, '$destroy');
rowScopes.length = 0;
if ($scope.indexPattern.sourceFilters) {
$scope.rows = [];
each($scope.indexPattern.sourceFilters, (filter) => {
const matcher = fieldWildcardMatcher([ filter.value ]);
// compute which fields match a filter
const matches = $scope.indexPattern.getNonScriptedFields().map(f => f.name).filter(matcher).sort();
if ($scope.$parent.fieldFilter && !angularFilter(matches, $scope.$parent.fieldFilter).length) {
return;
}
// compute the rows
const rowScope = $scope.$new();
rowScope.filter = filter;
rowScopes.push(rowScope);
$scope.rows.push([
{
markup: filterHtml,
scope: rowScope
},
size(matches) ? escape(matches.join(', ')) : '<em>The source filter doesn\'t match any known fields.</em>',
{
markup: controlsHtml,
scope: rowScope
}
]);
});
// Update the tab count
find($scope.$parent.editSections, {index: 'sourceFilters'}).count = $scope.rows.length;
}
});
}
all() {
return this.$scope.indexPattern.sourceFilters || [];
}
delete(filter) {
if (this.editing === filter) {
this.editing = null;
}
this.$scope.indexPattern.sourceFilters = without(this.all(), filter);
return this.save();
}
create() {
const value = this.newValue;
this.newValue = null;
this.$scope.indexPattern.sourceFilters = [...this.all(), { value }];
return this.save();
}
save() {
this.saving = true;
this.$scope.indexPattern.save()
.then(() => this.editing = null)
.catch(notify.error)
.finally(() => this.saving = false);
}
}
};
});

34
src/core_plugins/kibana/public/management/sections/indices/source_filters/source_filters.less Normal file
View file

@ -0,0 +1,34 @@
@import (reference) "~ui/styles/variables";
source-filters {
.header {
text-align: center;
}
.source-filters-container {
margin-top: 15px;
&.saving {
pointer-events: none;
opacity: .4;
transition: opacity 0.75s;
}
.source-filter {
display: flex;
align-items: center;
margin: 10px 0;
.value {
text-align: left;
flex: 1 1 auto;
padding-right: 5px;
font-family: @font-family-monospace;
:not(input) {
padding-left: 15px;
}
}
}
}
}

1
src/core_plugins/kibana/server/lib/schemas/resources/index_pattern_schema.js
View file

@ -6,6 +6,7 @@ module.exports = Joi.object({
time_field_name: Joi.string(),
interval_name: Joi.string(),
not_expandable: Joi.boolean(),
source_filters: Joi.array(),
fields: Joi.array().items(
Joi.object({
name: Joi.string().required(),

1
src/test_utils/stub_index_pattern.js
View file

@ -20,6 +20,7 @@ export default function (Private) {
this.timeFieldName = timeField;
this.getNonScriptedFields = sinon.spy();
this.getScriptedFields = sinon.spy();
this.getSourceFiltering = sinon.spy();
this.metaFields = ['_id', '_type', '_source'];
this.fieldFormatMap = {};
this.routes = IndexPattern.routes;

95
src/ui/public/courier/data_source/__tests__/search_source.js
View file

@ -4,17 +4,25 @@ import sinon from 'auto-release-sinon';
import RequestQueueProv from '../../_request_queue';
import SearchSourceProv from '../search_source';
import StubIndexPatternProv from 'test_utils/stub_index_pattern';
describe('SearchSource', function () {
require('test_utils/no_digest_promises').activateForSuite();
let requestQueue;
let SearchSource;
let indexPattern;
let indexPattern2;
beforeEach(ngMock.module('kibana'));
beforeEach(ngMock.inject(function (Private) {
requestQueue = Private(RequestQueueProv);
SearchSource = Private(SearchSourceProv);
const IndexPattern = Private(StubIndexPatternProv);
indexPattern = new IndexPattern('test-*', null, []);
indexPattern2 = new IndexPattern('test2-*', null, []);
expect(indexPattern).to.not.be(indexPattern2);
}));
describe('#onResults()', function () {
@ -56,4 +64,91 @@ describe('SearchSource', function () {
expect(requestQueue).to.have.length(0);
});
});
describe('#index()', function () {
describe('auto-sourceFiltering', function () {
context('new index pattern assigned', function () {
it('generates a source filter', function () {
const source = new SearchSource();
expect(source.get('index')).to.be(undefined);
expect(source.get('source')).to.be(undefined);
source.set('index', indexPattern);
expect(source.get('index')).to.be(indexPattern);
expect(source.get('source')).to.be.a('function');
});
it('removes created source filter on removal', function () {
const source = new SearchSource();
source.set('index', indexPattern);
source.set('index', null);
expect(source.get('index')).to.be(undefined);
expect(source.get('source')).to.be(undefined);
});
});
context('new index pattern assigned over another', function () {
it('replaces source filter with new', function () {
const source = new SearchSource();
source.set('index', indexPattern);
const sourceFilter1 = source.get('source');
source.set('index', indexPattern2);
expect(source.get('index')).to.be(indexPattern2);
expect(source.get('source')).to.be.a('function');
expect(source.get('source')).to.not.be(sourceFilter1);
});
it('removes created source filter on removal', function () {
const source = new SearchSource();
source.set('index', indexPattern);
source.set('index', indexPattern2);
source.set('index', null);
expect(source.get('index')).to.be(undefined);
expect(source.get('source')).to.be(undefined);
});
});
context('ip assigned before custom source filter', function () {
it('custom source filter becomes new source', function () {
const source = new SearchSource();
const football = {};
source.set('index', indexPattern);
expect(source.get('source')).to.be.a('function');
source.set('source', football);
expect(source.get('index')).to.be(indexPattern);
expect(source.get('source')).to.be(football);
});
it('custom source stays after removal', function () {
const source = new SearchSource();
const football = {};
source.set('index', indexPattern);
source.set('source', football);
source.set('index', null);
expect(source.get('index')).to.be(undefined);
expect(source.get('source')).to.be(football);
});
});
context('ip assigned after custom source filter', function () {
it('leaves the custom filter in place', function () {
const source = new SearchSource();
const football = {};
source.set('source', football);
source.set('index', indexPattern);
expect(source.get('index')).to.be(indexPattern);
expect(source.get('source')).to.be(football);
});
it('custom source stays after removal', function () {
const source = new SearchSource();
const football = {};
source.set('source', football);
source.set('index', indexPattern);
source.set('index', null);
expect(source.get('index')).to.be(undefined);
expect(source.get('source')).to.be(football);
});
});
});
});
});

9
src/ui/public/courier/data_source/_abstract.js
View file

@ -7,11 +7,13 @@ import RequestQueueProvider from '../_request_queue';
import ErrorHandlersProvider from '../_error_handlers';
import FetchProvider from '../fetch';
import DecorateQueryProvider from './_decorate_query';
import FieldWildcardProvider from '../../field_wildcard';
export default function SourceAbstractFactory(Private, Promise, PromiseEmitter) {
let requestQueue = Private(RequestQueueProvider);
let errorHandlers = Private(ErrorHandlersProvider);
let courierFetch = Private(FetchProvider);
let { fieldWildcardFilter } = Private(FieldWildcardProvider);
function SourceAbstract(initialState, strategy) {
let self = this;
@ -285,12 +287,17 @@ export default function SourceAbstractFactory(Private, Promise, PromiseEmitter)
if (flatState.body.size > 0) {
let computedFields = flatState.index.getComputedFields();
flatState.body.stored_fields = computedFields.storedFields;
flatState.body._source = computedFields._source;
flatState.body.script_fields = flatState.body.script_fields || {};
flatState.body.docvalue_fields = flatState.body.docvalue_fields || [];
_.extend(flatState.body.script_fields, computedFields.scriptFields);
flatState.body.docvalue_fields = _.union(flatState.body.docvalue_fields, computedFields.docvalueFields);
if (flatState.body._source) {
// exclude source fields for this index pattern specified by the user
const filter = fieldWildcardFilter(flatState.body._source.excludes);
flatState.body.docvalue_fields = flatState.body.docvalue_fields.filter(filter);
}
}
decorateQuery(flatState.body.query);

32
src/ui/public/courier/data_source/search_source.js
View file

@ -66,6 +66,12 @@ export default function SearchSourceFactory(Promise, Private, config) {
let searchStrategy = Private(SearchStrategyProvider);
let normalizeSortRequest = Private(NormalizeSortRequestProvider);
let forIp = Symbol('for which index pattern?');
function isIndexPattern(val) {
return Boolean(val && typeof val.toIndexList === 'function');
}
_.class(SearchSource).inherits(SourceAbstract);
function SearchSource(initialState) {
SearchSource.Super.call(this, initialState, searchStrategy);
@ -94,13 +100,31 @@ export default function SearchSourceFactory(Promise, Private, config) {
];
SearchSource.prototype.index = function (indexPattern) {
if (indexPattern === undefined) return this._state.index;
if (indexPattern === null) return delete this._state.index;
if (!indexPattern || typeof indexPattern.toIndexList !== 'function') {
let state = this._state;
let hasSource = state.source;
let sourceCameFromIp = hasSource && state.source.hasOwnProperty(forIp);
let sourceIsForOurIp = sourceCameFromIp && state.source[forIp] === state.index;
if (sourceIsForOurIp) {
delete state.source;
}
if (indexPattern === undefined) return state.index;
if (indexPattern === null) return delete state.index;
if (!isIndexPattern(indexPattern)) {
throw new TypeError('expected indexPattern to be an IndexPattern duck.');
}
this._state.index = indexPattern;
state.index = indexPattern;
if (!state.source) {
// imply source filtering based on the index pattern, but allow overriding
// it by simply setting another value for "source". When index is changed
state.source = function () {
return indexPattern.getSourceFiltering();
};
state.source[forIp] = indexPattern;
}
return this;
};

102
src/ui/public/field_wildcard/__tests__/field_wildcard.js Normal file
View file

@ -0,0 +1,102 @@
import expect from 'expect.js';
import ngMock from 'ng_mock';
import FieldWildcardProvider from '../../field_wildcard';
describe('fieldWildcard', function () {
let fieldWildcardFilter;
let makeRegEx;
beforeEach(ngMock.module('kibana'));
beforeEach(ngMock.inject(function (config, Private) {
config.set('metaFields', ['_id', '_type', '_source']);
const fieldWildcard = Private(FieldWildcardProvider);
fieldWildcardFilter = fieldWildcard.fieldWildcardFilter;
makeRegEx = fieldWildcard.makeRegEx;
}));
describe('makeRegEx', function () {
it('matches * in any position', function () {
expect('aaaaaabbbbbbbcccccc').to.match(makeRegEx('*a*b*c*'));
expect('a1234').to.match(makeRegEx('*1234'));
expect('1234a').to.match(makeRegEx('1234*'));
expect('12a34').to.match(makeRegEx('12a34'));
});
it('properly escapes regexp control characters', function () {
expect('account[user_id]').to.match(makeRegEx('account[*]'));
});
it('properly limits matches without wildcards', function () {
expect('username').to.match(makeRegEx('*name'));
expect('username').to.match(makeRegEx('user*'));
expect('username').to.match(makeRegEx('username'));
expect('username').to.not.match(makeRegEx('user'));
expect('username').to.not.match(makeRegEx('name'));
expect('username').to.not.match(makeRegEx('erna'));
});
});
describe('filter', function () {
it('filters nothing when given an empty array', function () {
const filter = fieldWildcardFilter([]);
const original = [
'foo',
'bar',
'baz',
1234
];
expect(original.filter(filter)).to.eql(original);
});
it('does not filter metaFields', function () {
const filter = fieldWildcardFilter([ '_*' ]);
const original = [
'_id',
'_type',
'_typefake'
];
expect(original.filter(filter)).to.eql(['_id', '_type']);
});
it('filters values that match the globs', function () {
const filter = fieldWildcardFilter([
'f*',
'*4'
]);
const original = [
'foo',
'bar',
'baz',
1234
];
expect(original.filter(filter)).to.eql(['bar', 'baz']);
});
it('handles weird values okay', function () {
const filter = fieldWildcardFilter([
'f*',
'*4',
'undefined'
]);
const original = [
'foo',
null,
'bar',
undefined,
{},
[],
'baz',
1234
];
expect(original.filter(filter)).to.eql([null, 'bar', {}, [], 'baz']);
});
});
});

28
src/ui/public/field_wildcard/field_wildcard.js Normal file
View file

@ -0,0 +1,28 @@
import { endsWith, escapeRegExp, memoize } from 'lodash';
export default function fieldWildcard(config) {
const metaFields = config.get('metaFields');
const makeRegEx = memoize(function makeRegEx(glob) {
return new RegExp('^' + glob.split('*').map(escapeRegExp).join('.*') + '$');
});
function fieldWildcardMatcher(globs) {
return function matcher(val) {
// do not test metaFields or keyword
if (metaFields.indexOf(val) !== -1) {
return false;
}
return globs.some(p => makeRegEx(p).test(val));
};
}
function fieldWildcardFilter(globs) {
const matcher = fieldWildcardMatcher(globs);
return function filter(val) {
return !matcher(val);
};
}
return { makeRegEx, fieldWildcardMatcher, fieldWildcardFilter };
};

4
src/ui/public/index_patterns/__tests__/_get_computed_fields.js
View file

@ -26,10 +26,6 @@ describe('get computed fields', function () {
expect(fn().storedFields).to.contain('*');
});
it('should request _source seperately', function () {
expect(fn()._source).to.be(true);
});
it('should request date fields as docvalue_fields', function () {
expect(fn().docvalueFields).to.contain('@timestamp');
expect(fn().docvalueFields).to.not.contain('bytes');

1
src/ui/public/index_patterns/_get_computed_fields.js
View file

@ -19,7 +19,6 @@ export default function () {
return {
storedFields: ['*'],
_source: true,
scriptFields: scriptFields,
docvalueFields: docvalueFields
};

11
src/ui/public/index_patterns/_index_pattern.js
View file

@ -33,7 +33,8 @@ export default function IndexPatternFactory(Private, Notifier, config, kbnIndex,
edit: '/management/kibana/indices/{{id}}',
addField: '/management/kibana/indices/{{id}}/create-field',
indexedFields: '/management/kibana/indices/{{id}}?_a=(tab:indexedFields)',
scriptedFields: '/management/kibana/indices/{{id}}?_a=(tab:scriptedFields)'
scriptedFields: '/management/kibana/indices/{{id}}?_a=(tab:scriptedFields)',
sourceFilters: '/management/kibana/indices/{{id}}?_a=(tab:sourceFilters)'
});
const mapping = mappingSetup.expandShorthand({
@ -42,6 +43,7 @@ export default function IndexPatternFactory(Private, Notifier, config, kbnIndex,
notExpandable: 'boolean',
intervalName: 'string',
fields: 'json',
sourceFilters: 'json',
fieldFormatMap: {
type: 'string',
_serialize(map = {}) {
@ -198,6 +200,13 @@ export default function IndexPatternFactory(Private, Notifier, config, kbnIndex,
.then(() => this);
}
// Get the source filtering configuration for that index.
getSourceFiltering() {
return {
excludes: this.sourceFilters && this.sourceFilters.map(filter => filter.value) || []
};
}
addScriptedField(name, script, type = 'string', lang) {
const scriptedFields = this.getScriptedFields();
const names = _.pluck(scriptedFields, 'name');

2
src/ui/public/watch_multi/watch_multi.js
View file

@ -25,7 +25,7 @@ uiModules.get('kibana')
* 4. a function that will be called, like a normal function water
*
* 5. an object with any of the properties:
* `get`: the getter called on each itteration
* `get`: the getter called on each iteration
* `deep`: a flag to turn on objectEquality in $watch
* `fn`: the watch registration function ($scope.$watch or $scope.$watchCollection)
*

2
test/fixtures/dump_data/dashboard.mapping.json vendored
View file

@ -1 +1 @@
{".kibana":{"mappings":{"config":{"properties":{"buildNum":{"type":"keyword"}}},"index-pattern":{"properties":{"fields":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"timeFieldName":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"title":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"search":{"properties":{"columns":{"type":"text"},"description":{"type":"text"},"hits":{"type":"integer"},"kibanaSavedObjectMeta":{"properties":{"searchSourceJSON":{"type":"text"}}},"sort":{"type":"text"},"title":{"type":"text"},"version":{"type":"integer"}}},"visualization":{"properties":{"description":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"kibanaSavedObjectMeta":{"properties":{"searchSourceJSON":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"title":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"uiStateJSON":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"version":{"type":"integer"},"visState":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"server":{"properties":{"uuid":{"type":"keyword"}}},"dashboard":{"properties":{"description":{"type":"text"},"hits":{"type":"integer"},"kibanaSavedObjectMeta":{"properties":{"searchSourceJSON":{"type":"text"}}},"optionsJSON":{"type":"text"},"panelsJSON":{"type":"text"},"timeFrom":{"type":"text"},"timeRestore":{"type":"boolean"},"timeTo":{"type":"text"},"title":{"type":"text"},"uiStateJSON":{"type":"text"},"version":{"type":"integer"}}}}}}
{".kibana":{"mappings":{"config":{"properties":{"buildNum":{"type":"keyword"}}},"index-pattern":{"properties":{"fields":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"timeFieldName":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"title":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"sourceFilters":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"search":{"properties":{"columns":{"type":"text"},"description":{"type":"text"},"hits":{"type":"integer"},"kibanaSavedObjectMeta":{"properties":{"searchSourceJSON":{"type":"text"}}},"sort":{"type":"text"},"title":{"type":"text"},"version":{"type":"integer"}}},"visualization":{"properties":{"description":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"kibanaSavedObjectMeta":{"properties":{"searchSourceJSON":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"title":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"uiStateJSON":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"version":{"type":"integer"},"visState":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"server":{"properties":{"uuid":{"type":"keyword"}}},"dashboard":{"properties":{"description":{"type":"text"},"hits":{"type":"integer"},"kibanaSavedObjectMeta":{"properties":{"searchSourceJSON":{"type":"text"}}},"optionsJSON":{"type":"text"},"panelsJSON":{"type":"text"},"timeFrom":{"type":"text"},"timeRestore":{"type":"boolean"},"timeTo":{"type":"text"},"title":{"type":"text"},"uiStateJSON":{"type":"text"},"version":{"type":"integer"}}}}}}

2
test/fixtures/dump_data/visualize.mapping.json vendored
View file

@ -1 +1 @@
{".kibana":{"mappings":{"index-pattern":{"properties":{"fields":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"timeFieldName":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"title":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}}}
{".kibana":{"mappings":{"index-pattern":{"properties":{"fields":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"timeFieldName":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"title":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"sourceFilters":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}}}

1
test/fixtures/dump_data/visualize_source-filters.data.json vendored Normal file
View file

File diff suppressed because one or more lines are too long

1
test/fixtures/dump_data/visualize_source-filters.mapping.json vendored Normal file
View file

@ -0,0 +1 @@
{".kibana":{"mappings":{"index-pattern":{"properties":{"fields":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"timeFieldName":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"title":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"sourceFilters":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}}}

51
test/functional/apps/discover/_source_filters.js Normal file
View file

@ -0,0 +1,51 @@
import {
bdd,
scenarioManager,
esClient,
elasticDump
} from '../../../support';
import PageObjects from '../../../support/page_objects';
var expect = require('expect.js');
bdd.describe('source filters', function describeIndexTests() {
bdd.before(function () {
var fromTime = '2015-09-19 06:31:44.000';
var toTime = '2015-09-23 18:31:44.000';
// delete .kibana index and update configDoc
return esClient.deleteAndUpdateConfigDoc({'dateFormat:tz':'UTC', 'defaultIndex':'logstash-*'})
.then(function loadkibanaIndexPattern() {
PageObjects.common.debug('load kibana index with default index pattern');
return elasticDump.elasticLoad('visualize_source-filters','.kibana');
})
// and load a set of makelogs data
.then(function loadIfEmptyMakelogs() {
return scenarioManager.loadIfEmpty('logstashFunctional');
})
.then(function () {
PageObjects.common.debug('discover');
return PageObjects.common.navigateToApp('discover');
})
.then(function () {
PageObjects.common.debug('setAbsoluteRange');
return PageObjects.header.setAbsoluteRange(fromTime, toTime);
})
.then(function () {
//After hiding the time picker, we need to wait for
//the refresh button to hide before clicking the share button
return PageObjects.common.sleep(1000);
});
});
bdd.it('should not get the field referer', function () {
return PageObjects.discover.getAllFieldNames()
.then(function (fieldNames) {
expect(fieldNames).to.not.contain('referer');
const relatedContentFields = fieldNames.filter((fieldName) => fieldName.indexOf('relatedContent') === 0);
expect(relatedContentFields).to.have.length(0);
});
});
});

1
test/functional/apps/discover/index.js
View file

@ -23,4 +23,5 @@ bdd.describe('discover app', function () {
require('./_field_data');
require('./_shared_links');
require('./_collapse_expand');
require('./_source_filters');
});

2
test/functional/apps/management/_index_pattern_create_delete.js
View file

@ -55,10 +55,10 @@ bdd.describe('creating and deleting default index', function describeIndexTests(
'searchable',
'aggregatable',
'analyzed',
'excluded',
'controls'
];
// 6 name type format analyzed indexed controls
expect(headers.length).to.be(expectedHeaders.length);
var comparedHeaders = headers.map(function compareHead(header, i) {

8
test/support/page_objects/discover_page.js
View file

@ -229,6 +229,14 @@ export default class DiscoverPage {
.click();
}
getAllFieldNames() {
return this.findTimeout
.findAllByClassName('sidebar-item')
.then((items) => {
return Promise.all(items.map((item) => item.getVisibleText()));
});
}
getSidebarWidth() {
return this.findTimeout
.findByClassName('sidebar-list')