mirror of
https://github.com/elastic/logstash.git
synced 2025-04-24 14:47:19 -04:00
parent
b19db959af
commit
c731596c7c
1 changed files with 41 additions and 14 deletions
55
docs/static/azure-module.asciidoc
vendored
55
docs/static/azure-module.asciidoc
vendored
|
@ -220,32 +220,59 @@ https://portal.azure.com[Azure Portal]`-> Blob Storage account -> Access keys`.
|
||||||
Here are some guidelines to help you achieve a successful deployment, and avoid
|
Here are some guidelines to help you achieve a successful deployment, and avoid
|
||||||
data conflicts that can cause lost events.
|
data conflicts that can cause lost events.
|
||||||
|
|
||||||
* **Create a {ls} consumer group.**
|
* <<azure-bp-group>>
|
||||||
|
* <<azure-bp-multihub>>
|
||||||
|
* <<azure-bp-threads>>
|
||||||
|
|
||||||
|
[[azure-bp-group]]
|
||||||
|
====== Create a {ls} consumer group
|
||||||
|
|
||||||
Create a new consumer group specifically for {ls}. Do not use the $default or
|
Create a new consumer group specifically for {ls}. Do not use the $default or
|
||||||
any other consumer group that might already be in use. Reusing consumer groups
|
any other consumer group that might already be in use. Reusing consumer groups
|
||||||
among non-related consumers can cause unexpected behavior and possibly lost
|
among non-related consumers can cause unexpected behavior and possibly lost
|
||||||
events. All {ls} instances should use the same consumer group so that they can
|
events. All {ls} instances should use the same consumer group so that they can
|
||||||
work together for processing events.
|
work together for processing events.
|
||||||
* **Avoid overwriting offset with multiple Event Hubs.**
|
|
||||||
|
[[azure-bp-multihub]]
|
||||||
|
====== Avoid overwriting offset with multiple Event Hubs
|
||||||
|
|
||||||
The offsets (position) of the Event Hubs are stored in the configured Azure Blob
|
The offsets (position) of the Event Hubs are stored in the configured Azure Blob
|
||||||
store. The Azure Blob store uses paths like a file system to store the offsets.
|
store. The Azure Blob store uses paths like a file system to store the offsets.
|
||||||
If the paths between multiple Event Hubs overlap, then the offsets may be stored
|
If the paths between multiple Event Hubs overlap, then the offsets may be stored
|
||||||
incorrectly.
|
incorrectly.
|
||||||
|
|
||||||
To avoid duplicate file paths, use the advanced configuration model and make
|
To avoid duplicate file paths, use the advanced configuration model and make
|
||||||
sure that at least one of these options is different per Event Hub:
|
sure that at least one of these options is different per Event Hub:
|
||||||
** storage_connection
|
|
||||||
** storage_container (defaults to Event Hub name if not defined)
|
* storage_connection
|
||||||
** consumer_group
|
* storage_container (defaults to Event Hub name if not defined)
|
||||||
* **Set number of threads correctly.**
|
* consumer_group
|
||||||
The number of threads should equal the number of Event Hubs plus one (or more).
|
|
||||||
Each Event Hub needs at least one thread. An additional thread is needed to help
|
[[azure-bp-threads]]
|
||||||
coordinate the other threads. The number of threads should not exceed the number of Event Hubs multiplied by the
|
====== Set number of threads correctly
|
||||||
number of partitions per Event Hub plus one. Threads are
|
|
||||||
currently available only as a global setting.
|
By default, the number of threads used to service all event hubs is `16`. And
|
||||||
** Sample: Event Hubs = 4. Partitions on each Event Hub = 3.
|
while this may be sufficient for most use cases, throughput may be improved by
|
||||||
|
refining this number. When servicing a large number of partitions across one or
|
||||||
|
more event hubs, setting a higher value may result in improved performance. The
|
||||||
|
maximum number of threads is not strictly bound by the total number of
|
||||||
|
partitions being serviced, but setting the value much higher than that may mean
|
||||||
|
that some threads are idle.
|
||||||
|
|
||||||
|
NOTE: The number of threads *must* be greater than or equal to the number of Event
|
||||||
|
hubs plus one.
|
||||||
|
|
||||||
|
NOTE: Threads are currently available only as a global setting across all event hubs
|
||||||
|
in a single `azure_event_hubs` input definition. However if your configuration
|
||||||
|
includes multiple `azure_event_hubs` inputs, the threads setting applies
|
||||||
|
independently to each.
|
||||||
|
|
||||||
|
**Sample scenarios:**
|
||||||
|
|
||||||
|
* Event Hubs = 4. Partitions on each Event Hub = 3.
|
||||||
Minimum threads is 5 (4 Event Hubs plus one). Maximum threads is 13 (4 Event
|
Minimum threads is 5 (4 Event Hubs plus one). Maximum threads is 13 (4 Event
|
||||||
Hubs times 3 partitions plus one).
|
Hubs times 3 partitions plus one).
|
||||||
** If you're collecting activity logs from only one specified event hub instance,
|
* If you're collecting activity logs from only one specified event hub instance,
|
||||||
then only 2 threads (1 Event Hub plus one) are required.
|
then only 2 threads (1 Event Hub plus one) are required.
|
||||||
|
|
||||||
[[azure-module-setup]]
|
[[azure-module-setup]]
|
||||||
|
@ -427,7 +454,7 @@ containers.
|
||||||
===== `threads`
|
===== `threads`
|
||||||
* Value type is <<number,number>>
|
* Value type is <<number,number>>
|
||||||
* Minimum value is `2`
|
* Minimum value is `2`
|
||||||
* Default value is `4`
|
* Default value is `16`
|
||||||
|
|
||||||
Total number of threads used to process events. The value you set here applies
|
Total number of threads used to process events. The value you set here applies
|
||||||
to all Event Hubs. Even with advanced configuration, this value is a global
|
to all Event Hubs. Even with advanced configuration, this value is a global
|
||||||
|
|
Loading…
Add table
Add a link
Reference in a new issue