[role="xpack"]
[[azure-module]]
=== Azure Module 
experimental[]

The https://azure.microsoft.com/en-us/overview/what-is-azure/[Microsoft Azure]
module in Logstash helps you easily integrate your Azure activity logs and SQL
diagnostic logs with the Elastic Stack. 

image::static/images/azure-flow.png["Azure Work Flow",width="80%"]

You can monitor your Azure cloud environments and SQL DB deployments with
deep operational insights across multiple Azure subscriptions. You can explore
the health of your infrastructure in real-time, accelerating root cause analysis
and decreasing overall time to resolution. The Azure module helps you: 

* Analyze infrastructure changes and authorization activity
* Identify suspicious behaviors and potential malicious actors
* Perform root-cause analysis by investigating user activity
* Monitor and optimize your SQL DB deployments.

NOTE: The Logstash Azure module is an
https://www.elastic.co/products/x-pack[{xpack}] feature under the Basic License
and is therefore free to use. Please contact
mailto:monitor-azure@elastic.co[monitor-azure@elastic.co] for questions or more
information.

The Azure module uses the
{logstash-ref}/plugins-inputs-azure_event_hubs.html[Logstash Azure Event Hubs
input plugin] to consume data from Azure Event Hubs. The module taps directly into the
Azure dashboard, parses and indexes events into Elasticsearch, and installs a
suite of {kib} dashboards to help you start exploring your data immediately.   

[[azure-dashboards]]
==== Dashboards

These {kib} dashboards are available and ready for you to use. You can use them as they are, or tailor them to meet your needs.

===== Infrastructure activity monitoring

* *Overview*. Top-level view into your Azure operations, including info about users, resource groups, service health, access, activities, and alerts. 

* *Alerts*. Alert info, including activity, alert status, and alerts heatmap

* *User Activity*. Info about system users, their activity, and requests.

===== SQL Database monitoring

* *SQL DB Overview*. Top-level view into your SQL Databases, including counts for databases, servers, resource groups, and subscriptions.

* *SQL DB Database View*. Detailed info about each SQL Database, including wait time, errors, DTU and storage utilization, size, and read and write input/output.

* *SQL DB Queries*. Info about SQL Database queries and performance. 

[[azure-module-prereqs]]
==== Prerequisites

Azure Monitor enabled with Azure Event Hubs and the Elastic Stack are required
for this module.

[[azure-elk-prereqs]]
===== Elastic prerequisites

The instructions below assume that you have {ls}, {es}, and {kib} running locally. 
You can also run {ls}, {es}, and {kib} on separate hosts. 

The Elastic Stack version 6.4 (or later) is required for this module. 

The Azure module uses the `azure_event_hubs` input plugin to consume logs and
metrics from your Azure environment. It is installed by default with {ls} 6.4
(or later). Basic understanding of the plugin and options is helpful when you
set up the Azure module.  
See the {logstash-ref}/plugins-inputs-azure_event_hubs.html[azure_event_hubs input
plugin documentation] for more information. 

Elastic products are https://www.elastic.co/downloads[available to download] and
easy to install. 

[[azure-prereqs]]
===== Azure prerequisites

Azure Monitor should be configured to stream logs to one or more Event Hubs. 
Logstash will need to access these Event Hubs instances to consume your Azure logs and metrics.
See <<azure-resources>> at the end of this topic for links to Microsoft Azure documentation.

[[configuring-azure]]
==== Configure the module

Specify <<azure_config_options,options>> for the Logstash Azure module in the
`logstash.yml` configuration file. 

* *Basic configuration.* You can use the `logstash.yml` file to configure inputs from multiple Event Hubs that share the same configuration.
Basic configuration is recommended for most use cases. 

* *Advanced configuration.*  The advanced configuration is available for deployments where different Event Hubs
require different configurations. The `logstash.yml` file holds your settings. Advanced configuration is not necessary or
recommended for most use cases. 

See the {logstash-ref}/plugins-inputs-azure_event_hubs.html[azure_event_hubs input plugin
documentation] for more information about basic and advanced configuration
models. 

[[basic-config-sample]]
===== Basic configuration sample

The configuration in the `logstash.yml` file is shared between Event Hubs.
Basic configuration is recommended for most use cases

["source","shell",subs="attributes"]
-----
modules:
  - name: azure
    var.elasticsearch.hosts: localhost:9200
    var.kibana.host: localhost:5601
    var.input.azure_event_hubs.consumer_group: "logstash" <1>
    var.input.azure_event_hubs.storage_connection: "DefaultEndpointsProtocol=https;AccountName=instance1..." <2>
    var.input.azure_event_hubs.threads: 9 <3>
    var.input.azure_event_hubs.event_hub_connections:
      - "Endpoint=sb://...EntityPath=insights-operational-logs" <4>
      - "Endpoint=sb://...EntityPath=insights-metrics-pt1m" <5>
      - "Endpoint=sb://...EntityPath=insights-logs-blocks"
      - "Endpoint=sb://...EntityPath=insights-logs-databasewaitstatistics"
      - "Endpoint=sb://...EntityPath=insights-logs-errors"
      - "Endpoint=sb://...EntityPath=insights-logs-querystoreruntimestatistics"
      - "Endpoint=sb://...EntityPath=insights-logs-querystorewaitstatistics"
      - "Endpoint=sb://...EntityPath=insights-logs-timeouts"
-----
<1> The `consumer_group` (optional) is highly recommended. See <<azure_best_practices>>.
<2> The `storage_connection` (optional) sets the Azure Blob Storage connection for tracking processing state for Event Hubs when scaling out a deployment with multiple Logstash instances. See <<scaling-blob>> for additional details.
<3> See <<azure_best_practices>> for guidelines on choosing an appropriate number of threads.
<4> This connection sets up the consumption of Activity Logs. By default, Azure Monitor uses the `insights-operational-logs` Event Hub name. Make sure this matches the name of the Event Hub specified for Activity Logs.
<5> This connection and the ones below set up the consumption of SQL DB diagnostic logs and metrics. By default, Azure Monitor uses all these different Event Hub names.

The basic configuration requires the `var.input.azure_event_hubs.` prefix
before a configuration option. 
Notice the notation for the `threads` option. 

[[adv-config-sample]]
===== Advanced configuration sample

Advanced configuration in the `logstash.yml` file supports Event Hub specific
options. Advanced configuration is available for more granular tuning of
threading and Blob Storage usage across multiple Event Hubs. Advanced
configuration is not necessary or recommended for most use cases. Use it only if
it is required for your deployment scenario. 

You must define the `header` array with `name` in the first position. You can
define other options in any order. The per Event Hub configuration takes
precedence. Any values not defined per Event Hub use the global config value.  

In this example `threads`, `consumer_group`, and `storage_connection` will be
applied to each of the configured Event Hubs. Note that `decorate_events` is
defined in both the global and per Event Hub configuration. The per Event Hub
configuration takes precedence, and the global configuration is effectively
ignored when the per Event Hub setting is present.

["source","shell",subs="attributes"]
-----
modules:
  - name: azure
    var.elasticsearch.hosts: localhost:9200
    var.kibana.host: localhost:5601
    var.input.azure_event_hubs.decorate_events: true <1>
    var.input.azure_event_hubs.threads: 9 <2>
    var.input.azure_event_hubs.consumer_group: "logstash"
    var.input.azure_event_hubs.storage_connection: "DefaultEndpointsProtocol=https;AccountName=instance1..."
    var.input.azure_event_hubs.event_hubs:
      - ["name",                                    "initial_position",  "storage_container",  "decorate_events",  "event_hub_connection"]                                   <3>
      - ["insights-operational-logs",                 "TAIL",              "activity-logs1",    "true",             "Endpoint=sb://...EntityPath=insights-operational-logs"]
      - ["insights-operational-logs",                 "TAIL",              "activity_logs2",  "true",             "Endpoint=sb://...EntityPath=insights-operational-logs"]   <4>  
      - ["insights-metrics-pt1m",                     "TAIL",              "dbmetrics",         "true",             "Endpoint=sb://...EntityPath=insights-metrics-pt1m"]
      - ["insights-logs-blocks",                      "TAIL",              "dbblocks",          "true",             "Endpoint=sb://...EntityPath=insights-logs-blocks"]
      - ["insights-logs-databasewaitstatistics",      "TAIL",              "dbwaitstats",       "false",            "Endpoint=sb://...EntityPath=insights-logs-databasewaitstatistics"]
      - ["insights-logs-errors",                      "HEAD",              "dberrors",          "true",             "Endpoint=sb://...EntityPath=insights-logs-errors"
      - ["insights-logs-querystoreruntimestatistics", "TAIL",              "dbstoreruntime",    "true",             "Endpoint=sb://...EntityPath=insights-logs-querystoreruntimestatistics"]
      - ["insights-logs-querystorewaitstatistics",    "TAIL",              "dbstorewaitstats",  "true",             "Endpoint=sb://...EntityPath=insights-logs-querystorewaitstatistics"]
      - ["insights-logs-timeouts",                    "TAIL",              "dbtimeouts",        "true",             "Endpoint=sb://...EntityPath=insights-logs-timeouts"]
-----
<1> You can specify global Event Hub options. They will be overridden by any configurations specified in the event_hubs option.
<2> See <<azure_best_practices>> for guidelines on choosing an appropriate number of threads.
<3> The header array must be defined with name in the first position. Other options can be defined in any order. The per Event Hub configuration takes precedence. Any values not defined per Event Hub use the global config value.
<4> This enables consuming from a second Activity Logs Event Hub that uses a different Blob Storage container. This is necessary to avoid the offsets from the first insights-operational-logs from overwriting the offsets for the second insights-operational-logs.

The advanced configuration doesn't require a prefix before a per Event Hub
configuration option. Notice the notation for the `initial_position` option. 

[[scaling-blob]]
===== Scale Event Hub consumption

An https://azure.microsoft.com/en-us/services/storage/blobs[Azure Blob Storage
account] is an essential part of Azure-to-Logstash configuration. 
It is required for users who want to scale out multiple {ls} instances to consume from Event Hubs.

A Blob Storage account is a central location that enables multiple instances of
{ls} to work together to process events. It records the
offset (location) of processed events. On restart, {ls} resumes processing
exactly where it left off.

Configuration notes:

*  A Blob Storage account is highly recommended for use with this module, and is
likely required for production servers.
* The `storage_connection` option passes the blob storage connection string. 
* Configure all {ls} instances to use the same `storage_connection` to get the
benefits of shared processing.

Sample Blob Storage connection string:

[source,text]
----
DefaultEndpointsProtocol=https;AccountName=logstash;AccountKey=ETOPnkd/hDAWidkEpPZDiXffQPku/SZdXhPSLnfqdRTalssdEuPkZwIcouzXjCLb/xPZjzhmHfwRCGo0SBSw==;EndpointSuffix=core.windows.net
----

Find the connection string to Blob Storage here: 
https://portal.azure.com[Azure Portal]`-> Blob Storage account -> Access keys`.

[[azure_best_practices]]
===== Best practices

Here are some guidelines to help you achieve a successful deployment, and avoid
data conflicts that can cause lost events.

* **Create a {ls} consumer group.** 
Create a new consumer group specifically for {ls}. Do not use the $default or
any other consumer group that might already be in use. Reusing consumer groups
among non-related consumers can cause unexpected behavior and possibly lost
events. All {ls} instances should use the same consumer group so that they can
work together for processing events.
* **Avoid overwriting offset with multiple Event Hubs.** 
The offsets (position) of the Event Hubs are stored in the configured Azure Blob
store. The Azure Blob store uses paths like a file system to store the offsets.
If the paths between multiple Event Hubs overlap, then the offsets may be stored
incorrectly. 
To avoid duplicate file paths, use the advanced configuration model and make
sure that at least one of these options is different per Event Hub:
** storage_connection
** storage_container (defaults to Event Hub name if not defined)
** consumer_group
* **Set number of threads correctly.** 
The number of threads should equal the number of Event Hubs plus one (or more).
Each Event Hub needs at least one thread. An additional thread is needed to help
coordinate the other threads. The number of threads should not exceed the number of Event Hubs multiplied by the
number of partitions per Event Hub plus one. Threads are
currently  available only as a global setting.
** Sample: Event Hubs = 4. Partitions on each Event Hub = 3.
Minimum threads is 5 (4 Event Hubs plus one). Maximum threads is 13 (4 Event
Hubs times 3 partitions plus one). 
** If you're collecting activity logs from only one specified event hub instance,
then only 2 threads (1 Event Hub plus one) are required.

[[azure-module-setup]]
==== Set up and run the module

Be sure that the `logstash.yml` file is <<configuring-azure,configured correctly>>. 

===== First time setup

Run this command from the Logstash directory:

["source","shell",subs="attributes"]
-----
bin/logstash --setup
-----

The `--modules azure` option starts a Logstash pipeline for ingestion from Azure
Event Hubs. The `--setup` option creates an `azure-*` index pattern in
Elasticsearch and imports Kibana dashboards and visualizations. 

===== Subsequent starts

Run this command from the Logstash directory:

["source","shell",subs="attributes"]
-----
bin/logstash 
-----

NOTE: The `--setup` option is intended only for first-time setup. If you include
`--setup` on subsequent runs, your existing Kibana dashboards will be
overwritten.

[[exploring-data-azure]]
==== Explore your data
When the Logstash Azure module starts receiving events, you can begin using the
packaged Kibana dashboards to explore and visualize your data. 

To explore your data with Kibana:

. Open a browser to http://localhost:5601[http://localhost:5601] (username:
  "elastic"; password: "{pwd}")
. Click *Dashboard*.
. Click *[Azure Monitor] Overview*.

[[azure_config_options]]
==== Configuration options

NOTE: All Event Hubs options are common to both basic and advanced
configurations, with the following exceptions. The basic configuration uses
`event_hub_connections` to support multiple connections. The advanced
configuration uses `event_hubs` and `event_hub_connection` (singular).

[[azure_event_hubs]]
===== `event_hubs`
* Value type is <<array,array>>
* No default value
* Ignored for basic and command line configuration
* Required for advanced configuration 

Defines the per Event Hubs configuration for the <<adv-config-sample,advanced configuration>>. 

The advanced configuration uses `event_hub_connection` instead of `event_hub_connections`. 
The `event_hub_connection` option is defined per Event Hub.

[[azure_event_hub_connections]]
===== `event_hub_connections`
* Value type is <<array,array>>
* No default value
* Required for basic and command line configuration
* Ignored for advanced configuration

List of connection strings that identifies the Event Hubs to be read. Connection
strings include the EntityPath for the Event Hub. 


[[azure_checkpoint_interval]]
===== `checkpoint_interval`
* Value type is <<number,number>>
* Default value is `5` seconds
* Set to `0` to disable.
  
Interval in seconds to write checkpoints during batch processing. Checkpoints
tell {ls} where to resume processing after a restart. Checkpoints are
automatically written at the end of each batch, regardless of this setting. 

Writing checkpoints too frequently can slow down processing unnecessarily. 


[[azure_consumer_group]]
===== `consumer_group`
* Value type is <<string,string>>
* Default value is `$Default` 

Consumer group used to read the Event Hub(s). Create a consumer group
specifically for Logstash. Then ensure that all instances of Logstash use that
consumer group so that they can work together properly.


[[azure_decorate_events]]
===== `decorate_events`

* Value type is <<boolean,boolean>>
* Default value is `false`

Adds metadata about the Event Hub, including `Event Hub name`, `consumer_group`,
`processor_host`, `partition`, `offset`, `sequence`, `timestamp`, and `event_size`.  


[[azure_initial_position]]
===== `initial_position`
* Value type is <<string,string>>
* Valid arguments are `beginning`, `end`, `look_back`
* Default value is `beginning`

When first reading from an Event Hub, start from this position: 

* `beginning` reads all pre-existing events in the Event Hub 
* `end` does not read any pre-existing events in the Event Hub 
* `look_back` reads `end` minus a number of seconds worth of pre-existing events.
You control the number of seconds using the `initial_position_look_back` option.

If `storage_connection` is set, the `initial_position` value is used only
the first time Logstash reads from the Event Hub.


[[azure_initial_position_look_back]]
===== `initial_position_look_back`
* Value type is <<number,number>>
* Default value is `86400`
* Used only if `initial_position` is set to `look-back`

Number of seconds to look back to find the initial position for pre-existing
events. This option is used only if `initial_position` is set to `look_back`. If
`storage_connection` is set, this configuration applies only the first time {ls}
reads from the Event Hub. 


[[azure_max_batch_size]]
===== `max_batch_size`

* Value type is <<number,number>>
* Default value is `125`

Maximum number of events retrieved and processed together. A checkpoint is
created after each batch. Increasing this value may help with performance, but
requires more memory. 


[[azure_storage_connection]]
===== `storage_connection`
* Value type is <<string,string>>
* No default value 

Connection string for blob account storage. Blob account storage persists the
offsets between restarts, and ensures that multiple instances of Logstash
process different partitions. 
When this value is set, restarts resume where processing left off.
When this value is not set, the `initial_position` value is used on every restart.

We strongly recommend that you define this value for production environments.


[[azure_storage_container]]
===== `storage_container`
* Value type is <<string,string>>
* Defaults to the Event Hub name if not defined

Name of the storage container used to persist offsets and allow multiple instances of {ls}
to work together. 
 
To avoid overwriting offsets, you can use different storage containers. This is
particularly important if you are monitoring two Event Hubs with the same name.
You can use the advanced configuration model to configure different storage
containers. 


[[azure_threads]]
===== `threads`
* Value type is <<number,number>>
* Minimum value is `2`
* Default value is `4`
  
Total number of threads used to process events. The value you set here applies
to all Event Hubs. Even with advanced configuration, this value is a global
setting, and can't be set per event hub. 

The number of threads should be the number of Event Hubs plus one or more. 
See <<azure_best_practices>> for more information.


include::shared-module-options.asciidoc[]
    
==== Azure module schema

This module reads data from the Azure Event Hub and adds some additional structure to the data for Activity Logs and SQL Diagnostics. The original data is always preserved and any data added or parsed will be namespaced under 'azure'. For example, 'azure.subscription' may have been parsed from a longer more complex URN.

[cols="<,<,<",options="header",]
|=======================================================================
|Name |Description|Notes

|azure.subscription 	|Azure subscription from which this data originates. 	|Some Activity Log events may not be associated with a subscription.
|azure.group 	|Primary type of data. |Current values are either 'activity_log' or 'sql_diagnostics' 	
|azure.category* 	|Secondary type of data specific to group from which the data originated 	|
|azure.provider 	|Azure provider 	| 
|azure.resource_group 	|Azure resource group 	| 
|azure.resource_type 	|Azure resource type 	| 
|azure.resource_name 	|Azure resource name 	| 
|azure.database 	|Azure database name, for display purposes 	|SQL Diagnostics only
|azure.db_unique_id 	|Azure database name that is guaranteed to be unique 	|SQL Diagnostics only
|azure.server 	|Azure server for the database 	|SQL Diagnostics only
|azure.server_and_database 	|Azure server and database combined 	|SQL Diagnostics only
|azure.metadata 	|Any @metadata added by the plugins, for example var.input.azure_event_hubs.decorate_events: true 	
|=======================================================================

Notes:

* Activity Logs can have the following categories: Administrative, ServiceHealth, Alert, Autoscale, Security
* SQL Diagnostics can have the following categories: Metric, Blocks, Errors, Timeouts, QueryStoreRuntimeStatistics, QueryStoreWaitStatistics, DatabaseWaitStatistics, SQLInsights

Microsoft documents Activity log schema
https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/monitoring-activity-log-schema[here].
The SQL Diagnostics data is documented
https://docs.microsoft.com/en-us/azure/sql-database/sql-database-metrics-diag-logging[here].
Elastic does not own these data models, and as such, cannot make any
assurances of information accuracy or passivity.

===== Special note - Properties field

Many of the logs contain a `properties` top level field. This is often where the
most interesting data lives. There is not a fixed schema between log types for
properties fields coming from different sources. 

For example, one log may have
`properties.type` where one log sets this a String type and another sets this an
Integer type. To avoid mapping errors, the original properties field is moved to
`<azure.group>_<azure_category>_properties.<original_key>`.  
For example
`properties.type` may end up as `sql_diagnostics_Errors_properties.type` or
`activity_log_Security_properties.type` depending on the group/category where
the event originated.

[[azure-production]]
==== Deploying the module in production 

Use security best practices to secure your configuration.
See {ref}/secure-cluster.html[Secure a cluster] for details and recommendations.

[[azure-resources]]
==== Microsoft Azure resources 

Microsoft is the best source for the most up-to-date Azure information. 

* https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/monitoring-overview-azure-monitor[Overview of Azure Monitor]
* https://docs.microsoft.com/en-us/azure/sql-database/sql-database-metrics-diag-logging[Azure SQL Database metrics and diagnostics logging]
* https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/monitoring-stream-activity-logs-event-hubs[Stream the Azure Activity Log to Event Hubs]

