mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-06-29 01:44:36 -04:00
0782ae7427
* [ML] Text/Log categorization multi-bucket aggregation (#71752) This commit adds a new multi-bucket aggregation: `categorize_text` The aggregation follows a similar design to significant text in that it reads from `_source` and re-analyzes the the text as it is read. Key difference is that it does not use the indexed field's analyzer, but instead relies on the `ml_standard` tokenizer with specialized ML token filters. The tokenizer + filters are the same that machine learning categorization anomaly jobs utilize. The high level logical flow is as follows: - at each shard, read in the text field with a custom analyzer using `ml_standard` tokenizer - Read in the particular tokens from the analyzer - Feed these tokens to a token tree algorithm (an adaptation of the drain categorization algorithm) - Gather the individual log categories (the leaf nodes), sort them by doc_count, ship those buckets to be merged - Merge all buckets that have the EXACT same key - Once all buckets are merged, pass those keys + counts to a new token tree for additional merging - That tree builds the final buckets and that is returned to the user Algorithm explanation: - Each log is parsed with the ml-standard tokenizer - each token is passed into a token tree - For `max_match_token` each token is stored in the tree and at `max_match_token+1` (or `len(tokens)`) a log group is created - If another log group exists at that leaf, merge it if they have `similarity_threshold` percentage of tokens in common - merging simply replaces tokens that are different in the group with `*` - If a layer in the tree has `max_unique_tokens` we add a `*` child and any new tokens are passed through there. Catch here is that on the final merge, we first attempt to merge together subtrees with the smallest number of documents. Especially if the new sub tree has more documents counted. ## Aggregation configuration. Here is an example on some openstack logs ```js POST openstack/_search?size=0 { "aggs": { "categories": { "categorize_text": { "field": "message", // The field to categorize "similarity_threshold": 20, // merge log groups if they are this similar "max_unique_tokens": 20, // Max Number of children per token position "max_match_token": 4, // Maximum tokens to build prefix trees "size": 1 } } } } ``` This will return buckets like ```json "aggregations" : { "categories" : { "buckets" : [ { "doc_count" : 806, "key" : "nova-api.log.1.2017-05-16_13 INFO nova.osapi_compute.wsgi.server * HTTP/1.1 status len time" } ] } } ``` * fixing for backport * fixing test after backport
75 lines
2.6 KiB
Text
75 lines
2.6 KiB
Text
[[search-aggregations-bucket]]
|
|
== Bucket aggregations
|
|
|
|
Bucket aggregations don't calculate metrics over fields like the metrics aggregations do, but instead, they create
|
|
buckets of documents. Each bucket is associated with a criterion (depending on the aggregation type) which determines
|
|
whether or not a document in the current context "falls" into it. In other words, the buckets effectively define document
|
|
sets. In addition to the buckets themselves, the `bucket` aggregations also compute and return the number of documents
|
|
that "fell into" each bucket.
|
|
|
|
Bucket aggregations, as opposed to `metrics` aggregations, can hold sub-aggregations. These sub-aggregations will be
|
|
aggregated for the buckets created by their "parent" bucket aggregation.
|
|
|
|
There are different bucket aggregators, each with a different "bucketing" strategy. Some define a single bucket, some
|
|
define fixed number of multiple buckets, and others dynamically create the buckets during the aggregation process.
|
|
|
|
NOTE: The <<search-settings-max-buckets,`search.max_buckets`>> cluster setting
|
|
limits the number of buckets allowed in a single response.
|
|
|
|
include::bucket/adjacency-matrix-aggregation.asciidoc[]
|
|
|
|
include::bucket/autodatehistogram-aggregation.asciidoc[]
|
|
|
|
include::bucket/categorize-text-aggregation.asciidoc[]
|
|
|
|
include::bucket/children-aggregation.asciidoc[]
|
|
|
|
include::bucket/composite-aggregation.asciidoc[]
|
|
|
|
include::bucket/datehistogram-aggregation.asciidoc[]
|
|
|
|
include::bucket/daterange-aggregation.asciidoc[]
|
|
|
|
include::bucket/diversified-sampler-aggregation.asciidoc[]
|
|
|
|
include::bucket/filter-aggregation.asciidoc[]
|
|
|
|
include::bucket/filters-aggregation.asciidoc[]
|
|
|
|
include::bucket/geodistance-aggregation.asciidoc[]
|
|
|
|
include::bucket/geohashgrid-aggregation.asciidoc[]
|
|
|
|
include::bucket/geotilegrid-aggregation.asciidoc[]
|
|
|
|
include::bucket/global-aggregation.asciidoc[]
|
|
|
|
include::bucket/histogram-aggregation.asciidoc[]
|
|
|
|
include::bucket/iprange-aggregation.asciidoc[]
|
|
|
|
include::bucket/missing-aggregation.asciidoc[]
|
|
|
|
include::bucket/multi-terms-aggregation.asciidoc[]
|
|
|
|
include::bucket/nested-aggregation.asciidoc[]
|
|
|
|
include::bucket/parent-aggregation.asciidoc[]
|
|
|
|
include::bucket/range-aggregation.asciidoc[]
|
|
|
|
include::bucket/rare-terms-aggregation.asciidoc[]
|
|
|
|
include::bucket/reverse-nested-aggregation.asciidoc[]
|
|
|
|
include::bucket/sampler-aggregation.asciidoc[]
|
|
|
|
include::bucket/significantterms-aggregation.asciidoc[]
|
|
|
|
include::bucket/significanttext-aggregation.asciidoc[]
|
|
|
|
include::bucket/terms-aggregation.asciidoc[]
|
|
|
|
include::bucket/variablewidthhistogram-aggregation.asciidoc[]
|
|
|
|
include::bucket/range-field-note.asciidoc[]
|