kibana/docs/user/ml/index.asciidoc

[role="xpack"]
[[xpack-ml]]
= {ml-cap}

[partintro]
--
As data sets increase in size and complexity, the human effort required to
inspect dashboards or maintain rules for spotting infrastructure problems,
cyber attacks, or business issues becomes impractical. Elastic {ml-features}
such as {anomaly-detect} and {oldetection} make it easier to notice suspicious
activities with minimal human interference.

{kib} includes a free *{data-viz}* to learn more about your data. In particular,
if your data is stored in {es} and contains a time field, you can use the
*{data-viz}* to identify possible fields for {anomaly-detect}:

[role="screenshot"]
image::user/ml/images/ml-data-visualizer-sample.png[{data-viz} for sample flight data]

You can also upload a CSV, NDJSON, or log file. The *{data-viz}*
identifies the file format and field mappings. You can then optionally import
that data into an {es} index. To change the default file size limit, see
<<kibana-general-settings, fileUpload:maxFileSize advanced settings>>.

If {stack-security-features} are enabled, users must have the necessary
privileges to use {ml-features}. Refer to
{ml-docs}/setup.html#setup-privileges[Set up {ml-features}].

NOTE: There are limitations in {ml-features} that affect {kib}. For more 
information, refer to {ml-docs}/ml-limitations.html[{ml-cap}].

--

[[xpack-ml-anomalies]]
== {anomaly-detect-cap}

The Elastic {ml} {anomaly-detect} feature automatically models the normal
behavior of your time series data — learning trends, periodicity, and more — in
real time to identify anomalies, streamline root cause analysis, and reduce
false positives. {anomaly-detect-cap} runs in and scales with {es}, and
includes an intuitive UI on the {kib} *Machine Learning* page for creating
{anomaly-jobs} and understanding results.

If you have a license that includes the {ml-features}, you can
create {anomaly-jobs} and manage jobs and {dfeeds} from the *Job Management*
pane:

[role="screenshot"]
image::user/ml/images/ml-job-management.png[Job Management]

You can use the *Settings* pane to create and edit calendars and the
filters that are used in custom rules:

[role="screenshot"]
image::user/ml/images/ml-settings.png[Calendar Management]

The *Anomaly Explorer* and *Single Metric Viewer* display the results of your
{anomaly-jobs}. For example:

[role="screenshot"]
image::user/ml/images/ml-single-metric-viewer.png[Single Metric Viewer]

You can optionally add annotations by drag-selecting a period of time in
the *Single Metric Viewer* and adding a description. For example, you can add an
explanation for anomalies in that time period or provide notes about what is
occurring in your operational environment at that time:

[role="screenshot"]
image::user/ml/images/ml-annotations-list.png[Single Metric Viewer with annotations]

In some circumstances, annotations are also added automatically. For example, if
the {anomaly-job} detects that there is missing data, it annotates the affected
time period. For more information, see
{ml-docs}/ml-delayed-data-detection.html[Handling delayed data]. The
*Job Management* pane shows the full list of annotations for each job.

NOTE: The {kib} {ml-features} use pop-ups. You must configure your web
browser so that it does not block pop-up windows or create an exception for your
{kib} URL.

For more information about the {anomaly-detect} feature, see
https://www.elastic.co/what-is/elastic-stack-machine-learning[{ml-cap} in the {stack}]
and {ml-docs}/ml-ad-overview.html[{ml-cap} {anomaly-detect}].

[[xpack-ml-dfanalytics]]
== {dfanalytics-cap}

The Elastic {ml} {dfanalytics} feature enables you to analyze your data using
{classification}, {oldetection}, and {regression} algorithms and generate new
indices that contain the results alongside your source data.

If you have a license that includes the {ml-features}, you can create
{dfanalytics-jobs} and view their results on the *Data Frame Analytics* page in
{kib}. For example:

[role="screenshot"]
image::user/ml/images/classification.png[{classification-cap} results in {kib}]

For more information about the {dfanalytics} feature, see
{ml-docs}/ml-dfanalytics.html[{ml-cap} {dfanalytics}].

[[xpack-ml-aiops]]
== AIOps Labs

AIOps Labs is a part of {ml-app} in {kib} which provides features that use 
advanced statistical methods to help you interpret your data and its behavior.

[discrete]
[[explain-log-rate-spikes]]
=== Explain log rate spikes

preview::[]

Explain log rate spikes is a feature that uses advanced statistical methods to 
identify reasons for increases in log rates. It makes it easy to find and 
investigate causes of unusual spikes by using the analysis workflow view. 
Examine the histogram chart of the log rates for a given {data-source}, and find 
the reason behind a particular change possibly in millions of log events across 
multiple fields and values.

You can find explain log rate spikes under **{ml-app}** > **AIOps Labs** where 
you can select the {data-source} or saved search that you want to analyze.

[role="screenshot"]
image::user/ml/images/ml-explain-log-rate-before.png[Log event histogram chart]

Select a spike in the log event histogram chart to start the analysis. It 
identifies statistically significant field-value combinations that contribute to 
the spike and displays them in a table. You can optionally choose to summarize 
the results into groups. The table also shows an indicator of the level of 
impact and a sparkline showing the shape of the impact in the chart. Hovering 
over a row displays the impact on the histogram chart in more detail. You can 
inspect a field in **Discover** by selecting this option under the **Actions** 
column. You can also pin a table row by clicking on it then move the cursor to 
the histogram chart. It displays a tooltip with exact count values for the 
pinned field which enables closer investigation.

Brushes in the chart show the baseline time range and the deviation in the 
analyzed data. You can move the brushes to redefine both the baseline and the 
deviation and rerun the analysis with the modified values.

[role="screenshot"]
image::user/ml/images/ml-explain-log-rate.png[Log rate spike explained]


[discrete]
[[log-pattern-analysis]]
=== Log pattern analysis

preview::[]

Log pattern analysis helps you to find patterns in unstructured log messages and 
makes it easier to examine your data. It performs categorization analysis on a 
selected field of a {data-source}, creates categories based on the data and 
displays them together with a chart that shows the distribution of each category 
and an example document that matches the category.

You can find log pattern analysis under **{ml-app}** > **AIOps Labs** where you 
can select the {data-source} or saved search that you want to analyze.

[role="screenshot"]
image::user/ml/images/ml-log-pattern-analysis.png[Log pattern analysis UI]

Select a field for categorization and optionally apply any filters that you 
want, then start the analysis. The analysis uses the same algorithms as a {ml} 
categorization job. The results of the analysis are shown in a table that makes 
it possible to open **Discover** and show or filter out the given category 
there, which helps you to further examine your log messages.