kibana/docs/management/rollups/create_and_manage_rollups.asciidoc

[role="xpack"]
[[data-rollups]]
== Rollup Jobs

deprecated::[8.11.0,'Rollups are deprecated and will be removed in a future version. Use {ref}/downsampling.html[downsampling] instead.']

A rollup job is a periodic task that aggregates data from indices specified by
an index pattern, and then rolls it into a new index. Rollup indices are a good
way to compactly store months or years of historical data for use in
visualizations and reports.

You can go to the *Rollup Jobs* page using the navigation menu or the 
<<kibana-navigation-search,global search field>>.

[role="screenshot"]
image::images/management_rollup_list.png[List of currently active rollup jobs]

Before using this feature, you should be familiar with how rollups work.
{ref}/xpack-rollup.html[Rolling up historical data] is a good source for more
detailed information.

[float]
=== Required permissions

The `manage_rollup` cluster privilege is required to access *Rollup jobs*.

To add the privilege, go to the *Roles* management page using the navigation menu or the 
<<kibana-navigation-search,global search field>>.

[float]
[[create-and-manage-rollup-job]]
=== Create a rollup job

{kib} makes it easy for you to create a rollup job by walking you through
the process. You fill in the name, data flow, and how often you want to roll
up the data.  Then you define a date histogram aggregation for the rollup job
and optionally define terms, histogram, and metrics aggregations.

When defining the index pattern, you must enter a name that is different than
the output rollup index. Otherwise, the job will attempt to capture the data in
the rollup index. For example, if your index pattern is `metricbeat-*`, you can
name your rollup index `rollup-metricbeat`, but not `metricbeat-rollup`.

[role="screenshot"]
image::images/management_create_rollup_job.png[Wizard that walks you through creation of a rollup job]

[float]
[[manage-rollup-job]]
=== Start, stop, and delete rollup jobs

Once you've saved a rollup job, you'll see it the *Rollup Jobs* overview page,
where you can drill down for further investigation. The *Manage* menu enables
you to start, stop, and delete the rollup job.
You must first stop a rollup job before deleting it.

[role="screenshot"]
image::images/management_rollup_job_details.png[Rollup job details]

You can't change a rollup job after you've created it. To select additional
fields or redefine terms, you must delete the existing job, and then create a
new one with the updated specifications. Be sure to use a different name for the
new rollup job&mdash;reusing the same name can lead to problems with mismatched
job configurations. Refer to {ref}/rollup-job-config.html[rollup job
configuration].

[float]
[[rollup-data-tutorial]]
=== Try it: Create and visualize rolled up data

This example creates a rollup job to capture log data from sample web logs.
Before you start, <<add-sample-data,add the web logs sample data set>>.

In this example, you want data that is older than 7 days in the
`kibana_sample_data_logs` index to roll up into the `rollup_logstash` index.
You'll bucket the rolled up data on an hourly basis, using `60m` for the time
bucket configuration.

For this example, the job will perform the rollup every minute. However, you'd
typically roll up less frequently in production.

[float]
==== Create the rollup job

As you walk through the *Create rollup job* UI, enter the data:

|===
|*Field* |*Value*

|Name
|`logs_job`

|Index pattern
|`kibana_sample_data_logs`

|Rollup index name
|`rollup_logstash`

|Frequency
|Every minute

|Page size
|1000

|Latency buffer
|7d

|Date field
|@timestamp

|Time bucket size
|60m

|Time zone
|UTC

|Terms
|geo.src, machine.os.keyword

|Histogram
|bytes, memory

|Histogram interval
|1000

|Metrics
|bytes (average)
|===

On the **Review and save** page, click **Start job now** and **Save**.

The terms, histogram, and metrics fields reflect
the key information to retain in the rolled up data: where visitors are from (geo.src),
what operating system they are using (machine.os.keyword),
and how much data is being sent (bytes).

You can now use the rolled up data for analysis at a fraction of the storage
cost of the original index. The original data can live side by side with the new
rollup index, or you can remove or archive it using
{ref}/index-lifecycle-management.html[{ilm} ({ilm-init})].

[float]
==== Visualize the rolled up data

Your next step is to visualize your rolled up data in a vertical bar chart.
Most visualizations support rolled up data, with the exception of Timelion and Vega visualizations.

. Go to the *Data Views* page using the navigation menu or the 
<<kibana-navigation-search,global search field>>.

. Click *Create data view*, and select *Rollup data view* from the dropdown.

. Enter *rollup_logstash,kibana_sample_logs* as your *Data View* and `@timestamp`
as the *Time Filter field name*.
+
The notation for a combination data view with both raw and rolled up data is
`rollup_logstash,kibana_sample_data_logs`. In this data view, `rollup_logstash`
matches the rollup index and `kibana_sample_data_logs` matches the raw data.

. Go to *Dashboards*, then select *Create dashboard*.

. Set the <<set-time-filter,time filter>> to *Last 90 days*.

. On the dashboard, click *Create visualization*.

. Choose `rollup_logstash,kibana_sample_data_logs`
as your source to see both the raw and rolled up data.
+
[role="screenshot"]
image::images/management-create-rollup-bar-chart.png[Create visualization of rolled up data]

. Select *Bar* in the chart type dropdown.

. Add the `@timestamp` field to the *Horizontal axis*.

. Add the `bytes` field to the *Vertical axis*, defaulting to an `Average of
bytes`.
+
{kib} creates a vertical bar chart of your data. Select a section of the chart
to zoom in.
+
[role="screenshot"]
image::images/management_rollup_job_dashboard.png[Dashboard with rolled up data]