[[deploying-and-scaling]] == Deploying and Scaling Logstash The Elastic Stack is used for tons of use cases, from operational log and metrics analytics, to enterprise and application search. Making sure your data gets scalably, durably, and securely transported to Elasticsearch is extremely important, especially for mission critical environments. The goal of this document is to highlight the most common architecture patterns for Logstash and how to effectively scale as your deployment grows. The focus will be around the operational log, metrics, and security analytics use cases because they tend to require larger scale deployments. The deploying and scaling recommendations provided here may vary based on your own requirements. [float] [[deploying-getting-started]] === Getting Started For first time users, if you simply want to tail a log file to grasp the power of the Elastic Stack, we recommend trying {filebeat-ref}/filebeat-modules-overview.html[Filebeat Modules]. Filebeat Modules enable you to quickly collect, parse, and index popular log types and view pre-built Kibana dashboards within minutes. {metricbeat-ref}/metricbeat-modules.html[Metricbeat Modules] provide a similar experience, but with metrics data. In this context, Beats will ship data directly to Elasticsearch where {ref}/ingest.html[Ingest Nodes] will process and index your data. image::static/images/deploy1.png[] [float] ==== Introducing Logstash What are the main benefits for integrating Logstash into your architecture? * Scale through ingestion spikes - Logstash has an adaptive disk-based buffering system that will absorb incoming throughput, therefore mitigating backpressure * Ingest from other data sources like databases, S3, or messaging queues * Emit data to multiple destinations like S3, HDFS, or write to a file * Compose more sophisticated processing pipelines with conditional dataflow logic [float] [[scaling-ingest]] === Scaling Ingest Beats and Logstash make ingest awesome. Together, they provide a comprehensive solution that is scalable and resilient. What can you expect? * Horizontal scalability, high availability, and variable load handling * Message durability with at-least-once delivery guarantees * End-to-end secure transport with authentication and wire encryption [float] ==== Beats and Logstash Beats run across thousands of edge host servers, collecting, tailing, and shipping logs to Logstash. Logstash serves as the centralized streaming engine for data unification and enrichment. The <> exposes a secure, acknowledgement-based endpoint for Beats to send data to Logstash. image::static/images/deploy2.png[] NOTE: Enabling persistent queues is strongly recommended, and these architecture characteristics assume that they are enabled. We encourage you to review the <> documentation for feature benefits and more details on resiliency. [float] ==== Scalability Logstash is horizontally scalable and can form groups of nodes running the same pipeline. Logstash’s adaptive buffering capabilities will facilitate smooth streaming even through variable throughput loads. If the Logstash layer becomes an ingestion bottleneck, simply add more nodes to scale out. Here are a few general recommendations: * Beats should {filebeat-ref}/load-balancing.html[load balance] across a group of Logstash nodes. * A minimum of two Logstash nodes are recommended for high availability. * It’s common to deploy just one Beats input per Logstash node, but multiple Beats inputs can also be deployed per Logstash node to expose independent endpoints for different data sources. [float] ==== Resiliency When using https://www.elastic.co/products/beats/filebeat[Filebeat] or https://www.elastic.co/products/beats/winlogbeat[Winlogbeat] for log collection within this ingest flow, *at-least-once delivery* is guaranteed. Both the communication protocols, from Filebeat or Winlogbeat to Logstash, and from Logstash to Elasticsearch, are synchronous and support acknowledgements. The other Beats don’t yet have support for acknowledgements. Logstash persistent queues provide protection across node failures. For disk-level resiliency in Logstash, it’s important to ensure disk redundancy. For on-premise deployments, it's recommended that you configure RAID. When running in the cloud or a containerized environment, it’s recommended that you use persistent disks with replication strategies that reflect your data SLAs. NOTE: Make sure `queue.checkpoint.writes: 1` is set for at-least-once guarantees. For more details, see the <> documentation. [float] ==== Processing Logstash will commonly extract fields with <> or <>, augment <> info, and can further enrich events with <>, <>, or <> lookup datasets. Be aware that processing complexity can affect overall throughput and CPU utilization. Make sure to check out the other <>. [float] ==== Secure Transport Enterprise-grade security is available across the entire delivery chain. * Wire encryption is recommended for both the transport from {filebeat-ref}/configuring-ssl-logstash.html[Beats to Logstash] and from {logstash-ref}/ls-security.html[Logstash to Elasticsearch]. * There’s a wealth of security options when communicating with Elasticsearch including basic authentication, TLS, PKI, LDAP, AD, and other custom realms. To enable Elasticsearch security, see {ref}/secure-cluster.html[Secure a cluster]. [float] ==== Monitoring When running Logstash 5.2 or greater, the https://www.elastic.co/products/x-pack/monitoring[Monitoring UI] provides deep visibility into your deployment metrics, helping observe performance and alleviate bottlenecks as you scale. Monitoring is an X-Pack feature under the Basic License and is therefore *free to use*. To get started, see {logstash-ref}/monitoring-logstash.html[Monitoring Logstash]. If external monitoring is preferred, there are <> that return point-in-time metrics snapshots. [float] [[adding-other-sources]] === Adding Other Popular Sources Users may have other mechanisms of collecting logging data, and it’s easy to integrate and centralize them into the Elastic Stack. Let’s walk through a few scenarios: image::static/images/deploy3.png[] [float] ==== TCP, UDP, and HTTP Protocols The TCP, UDP, and HTTP protocols are common ways to feed data into Logstash. Logstash can expose endpoint listeners with the respective <>, <>, and <> input plugins. The data sources enumerated below are typically ingested through one of these three protocols. NOTE: The TCP protocol does not support application-level acknowledgements, so connectivity issues may result in data loss. For high availability scenarios, a third-party hardware or software load balancer, like HAProxy, should be added to fan out traffic to a group of Logstash nodes. [float] ==== Network and Security Data Although Beats may already satisfy your data ingest use case, network and security datasets come in a variety of forms. Let’s touch on a few other ingestion points. * Network wire data - collect and analyze network traffic with https://www.elastic.co/products/beats/packetbeat[Packetbeat]. * Netflow v5/v9/v10 - Logstash understands data from Netflow/IPFIX exporters with the <>. * Nmap - Logstash accepts and parses Nmap XML data with the <>. * SNMP trap - Logstash has a native <>. * CEF - Logstash accepts and parses CEF data from systems like Arcsight SmartConnectors with the <>. See this https://www.elastic.co/blog/integrating-elastic-stack-with-arcsight-siem-part-1[blog series] for more details. [float] ==== Centralized Syslog Servers Existing syslog server technologies like rsyslog and syslog-ng generally send syslog over to Logstash TCP or UDP endpoints for extraction, processing, and persistence. If the data format conforms to RFC3164, it can be fed directly to the <>. [float] ==== Infrastructure & Application Data and IoT Infrastructure and application metrics can be collected with https://www.elastic.co/products/beats/metricbeat[Metricbeat], but applications can also send webhooks to a Logstash HTTP input or have metrics polled from an HTTP endpoint with the <>. For applications that log with log4j2, it’s recommended to use the SocketAppender to send JSON to the Logstash TCP input. Alternatively, log4j2 can also log to a file for collection with FIlebeat. Usage of the log4j1 SocketAppender is not recommended. IoT devices like Raspberry Pis, smartphones, and connected vehicles often send telemetry data through one of these protocols. [float] [[integrating-with-messaging-queues]] === Integrating with Messaging Queues If you are leveraging message queuing technologies as part of your existing infrastructure, getting that data into the Elastic Stack is easy. For existing users who are utilizing an external queuing layer like Redis or RabbitMQ just for data buffering with Logstash, it’s recommended to use Logstash persistent queues instead of an external queuing layer. This will help with overall ease of management by removing an unnecessary layer of complexity in your ingest architecture. For users who want to integrate data from existing Kafka deployments or require the underlying usage of ephemeral storage, Kafka can serve as a data hub where Beats can persist to and Logstash nodes can consume from. image::static/images/deploy4.png[] The other TCP, UDP, and HTTP sources can persist to Kafka with Logstash as a conduit to achieve high availability in lieu of a load balancer. A group of Logstash nodes can then consume from topics with the <> to further transform and enrich the data in transit. [float] ==== Resiliency and Recovery When Logstash consumes from Kafka, persistent queues should be enabled and will add transport resiliency to mitigate the need for reprocessing during Logstash node failures. In this context, it’s recommended to use the default persistent queue disk allocation size `queue.max_bytes: 1GB`. If Kafka is configured to retain data for an extended period of time, data can be reprocessed from Kafka in the case of disaster recovery and reconciliation. [float] ==== Other Messaging Queue Integrations Although an additional queuing layer is not required, Logstash can consume from a myriad of other message queuing technologies like <> and <>. It also supports ingestion from hosted queuing services like <>, <>, and <>.