mirror of
https://github.com/elastic/logstash.git
synced 2025-04-24 06:37:19 -04:00
DeDe Morton
parent
1af8071b6b
commit
7a40590dae
2 changed files with 407 additions and 0 deletions
|
|
@ -33,6 +33,8 @@ include::static/config-management.asciidoc[]
|
|||
|
||||
include::static/modules.asciidoc[]
|
||||
|
||||
include::static/arcsight-module.asciidoc[]
|
||||
|
||||
// Working with Filebeat Modules
|
||||
|
||||
:edit_url: https://github.com/elastic/logstash/edit/master/docs/static/filebeat-modules.asciidoc
|
||||
|
|
|
|||
405
docs/static/arcsight-module.asciidoc
vendored
Normal file
405
docs/static/arcsight-module.asciidoc
vendored
Normal file
|
|
@ -0,0 +1,405 @@
|
|||
[role="xpack"]
|
||||
[[arcsight-module]]
|
||||
=== Logstash ArcSight Module
|
||||
|
||||
++++
|
||||
<titleabbrev>ArcSight Module</titleabbrev>
|
||||
++++
|
||||
|
||||
NOTE: The Logstash ArcSight module is an
|
||||
https://www.elastic.co/products/x-pack[X-Pack] feature under the Basic License
|
||||
and is therefore free to use. Please contact
|
||||
mailto:arcsight@elastic.co[arcsight@elastic.co] for questions or more
|
||||
information.
|
||||
|
||||
The Logstash ArcSight module enables you to easily integrate with and augment
|
||||
the ArcSight Data Platform (ADP) to explore your data in Elasticsearch using
|
||||
Kibana visualizations. With a single command, the module taps directly into the
|
||||
ADP data stream, parses and indexes the security events into Elasticsearch, and
|
||||
installs a suite of Kibana dashboards to get you exploring your data
|
||||
immediately.
|
||||
|
||||
[role="screenshot"]
|
||||
image::static/images/arcsight-architecture-diagram-2017.png[ArcSight architecture]
|
||||
|
||||
[[arcsight-architecture]]
|
||||
==== Deployment Architecture
|
||||
|
||||
The Logstash ArcSight module understands CEF (Common Event Format), and can
|
||||
accept, enrich, and index these events for analysis on the Elastic Stack. ADP
|
||||
contains two core data collection components for data streaming:
|
||||
|
||||
* The _EventBroker (EB)_ is the central hub for incoming data and is based on
|
||||
open source Apache Kafka. The Logstash ArcSight module can consume directly from
|
||||
EB topics.
|
||||
* The _SmartConnectors (SC)_ are edge log collectors that parse and normalize
|
||||
data to CEF prior to publishing to the EB.
|
||||
|
||||
[[arcsight-getting-started]]
|
||||
==== Getting Started
|
||||
|
||||
To get started, you can use a basic Elastic Stack setup that reads events from
|
||||
the EB event stream.
|
||||
|
||||
[[arcsight-requirements]]
|
||||
===== Requirements
|
||||
|
||||
* These instructions assume you have the Elastic Stack (Logstash, Elasticsearch,
|
||||
Kibana) already installed. The products you need are
|
||||
https://www.elastic.co/downloads[available to download] and easy to install. The
|
||||
Elastic Stack 5.6 or higher is required for this module.
|
||||
* The Elastic Stack is running locally with default ports exposed, namely
|
||||
Elasticsearch as “localhost:9200” and Kibana as “localhost:5601”.
|
||||
* By default, the Logstash ArcSight module consumes from the EB “eb-cef” topic.
|
||||
For additional EB settings, see <<arcsight-module-config>>. Consuming from a
|
||||
secured EB port is not currently available.
|
||||
|
||||
[[arcsight-instructions]]
|
||||
===== Instructions
|
||||
|
||||
. {ref}/installing-xpack-es.html[Install X-Pack on Elasticsearch] and then start
|
||||
Elasticsearch.
|
||||
|
||||
. {kibana-ref}/installing-xpack-kb.html[Install X-Pack on Kibana] and then start
|
||||
Kibana.
|
||||
|
||||
. {logstash-ref}/installing-xpack-log.html[Install X-Pack on Logstash], which
|
||||
includes the Logstash ArcSight module. Then update the Logstash
|
||||
<<plugins-inputs-kafka,Kafka input plugin>> to an EB compatible version. In the
|
||||
Logstash install directory, run:
|
||||
+
|
||||
[source,shell]
|
||||
-----
|
||||
bin/logstash-plugin install x-pack
|
||||
bin/logstash-plugin install --version 6.2.7 logstash-input-kafka
|
||||
-----
|
||||
|
||||
. Start the Logstash ArcSight module by running the following command in the
|
||||
Logstash install directory with your respective EB host and port:
|
||||
+
|
||||
[source,shell]
|
||||
-----
|
||||
bin/logstash --modules arcsight --setup
|
||||
-M "arcsight.var.input.eventbroker.bootstrap_servers={eb_host}:{eb_port}"
|
||||
-M "arcsight.var.elasticsearch.username=elastic"
|
||||
-M "arcsight.var.elasticsearch.password=changeme"
|
||||
-M "arcsight.var.kibana.username=elastic"
|
||||
-M "arcsight.var.kibana.password=changeme"
|
||||
-----
|
||||
+
|
||||
--
|
||||
TIP: The command in this example is formatted for readability. Remove the line
|
||||
breaks before running this command.
|
||||
|
||||
The `--modules arcsight` option spins up an ArcSight CEF-aware Logstash
|
||||
pipeline for ingestion. The `--setup` option creates an `arcsight-*` index
|
||||
pattern in Elasticsearch and imports Kibana dashboards and visualizations. On
|
||||
subsequent module runs or when scaling out the Logstash deployment,
|
||||
the `--setup` option should be omitted to avoid overwriting the existing Kibana
|
||||
dashboards.
|
||||
--
|
||||
|
||||
. Explore your data with Kibana:
|
||||
.. Open browser @ http://localhost:5601[http://localhost:5601] (username:
|
||||
“elastic”; password: “changeme”)
|
||||
.. Open the “[ArcSight] Network Overview Dashboard”
|
||||
.. See <<exploring-data-arcsight>> for additional details on data exploration.
|
||||
|
||||
See <<configuring-arcsight>> if you want to specify additional options that
|
||||
control the behavior of the ArcSight module.
|
||||
|
||||
//ALVIN: All the config information needs to be together in one place, so I've added the above link and moved the section called "Configuring the Module" so that it appears along with the config options.
|
||||
|
||||
[[exploring-data-arcsight]]
|
||||
==== Exploring Your Security Data
|
||||
Once the Logstash ArcSight module starts receiving events, you can immediately
|
||||
begin using the packaged Kibana dashboards to explore and visualize your
|
||||
security data. The dashboards rapidly accelerate the time and effort required
|
||||
for security analysts and operators to gain situational and behavioral insights
|
||||
on network, endpoint, and DNS events flowing through the environment. You can
|
||||
use the dashboards as-is, or tailor them to work better with existing use cases
|
||||
and business requirements.
|
||||
|
||||
The dashboards have a navigation pane for context switching and drill downs
|
||||
across three core use cases:
|
||||
|
||||
* *Network Data*
|
||||
** Dashboards: Network Overview, Network Suspicious Activity
|
||||
** Data Types: Network firewalls, intrusion systems, VPN devices
|
||||
|
||||
* *Endpoint Data*
|
||||
** Dashboards: Endpoint Overview, Endpoint OS Activity
|
||||
** Data Types: Operating systems, applications, host intrusion systems
|
||||
|
||||
* *DNS Data*
|
||||
** Dashboards: Microsoft DNS Overview
|
||||
** Data Types: Microsoft DNS devices
|
||||
|
||||
[[network-dashboards-arsight]]
|
||||
===== Example Network Dashboards
|
||||
|
||||
[role="screenshot"]
|
||||
image::static/images/arcsight-network-overview.png[Network overview dashboard]
|
||||
|
||||
[role="screenshot"]
|
||||
image::static/images/arcsight-network-suspicious.png[Network suspicious activity dashboard]
|
||||
|
||||
These Kibana visualizations enable you to quickly understand the top devices,
|
||||
endpoints, attackers, and targets. This insight, along with the ability to
|
||||
instantly drill down on a particular host, port, device, or time range, offers a
|
||||
holistic view across the entire environment to identify specific segments that
|
||||
may require immediate attention or action. You can easily discover answers to
|
||||
questions like:
|
||||
|
||||
* Who are my attackers and what are they targeting?
|
||||
* Which of my devices or endpoints are the busiest and what services were
|
||||
rendered?
|
||||
* How many unique attackers, techniques, signatures, or targets were triggered
|
||||
at any given point in time?
|
||||
* What are the top sources, destinations, protocols, and behaviors that are
|
||||
causing the elevated count of failures?
|
||||
|
||||
[[configuring-arcsight]]
|
||||
==== Configuring the Module
|
||||
|
||||
You can specify additional options for the Logstash ArcSight module in the
|
||||
`logstash.yml` configuration file or with overrides through the command line
|
||||
like in the getting started. For more information about configuring modules, see
|
||||
<<logstash-modules>>.
|
||||
|
||||
As an example, the following settings can be appended to `logstash.yml` to
|
||||
configure your module:
|
||||
|
||||
[source,yaml]
|
||||
-----
|
||||
modules:
|
||||
- name: arcsight
|
||||
var.input.eventbroker.bootstrap_servers: “eb_host:39092”
|
||||
var.input.eventbroker.topics: “eb_topic”
|
||||
var.output.elasticsearch.hosts: "localhost:9200"
|
||||
var.elasticsearch.username: "elastic"
|
||||
var.elasticsearch.password: "changeme"
|
||||
var.kibana.host: “localhost:5601”
|
||||
var.kibana.username: "elastic"
|
||||
var.kibana.password: "changeme"
|
||||
-----
|
||||
|
||||
[[arcsight-module-config]]
|
||||
===== Logstash ArcSight Module Configuration Options
|
||||
|
||||
These are the configurable settings available for the Logstash ArcSight module.
|
||||
When overriding settings in the command line, the setting option must be
|
||||
prefixed with the module name, i.e. `arcsight.var.inputs` instead of `var.inputs`.
|
||||
|
||||
All settings are optional. If you don't specify configuration settings, Logstash
|
||||
uses the defaults.
|
||||
|
||||
//ALVIN: I don't see a good reason to have the summary table plus the reference info, so I have not included the table. It doesn't add enough value to warrant the real estate it takes up. Plus...tables don't work well when we go from docbook > PDF. If you feel strongly about having it, you can submit a separate PR, but make sure you make each setting an active link, or users will be extra annoyed by the repeated info. I've modified the intro text slightly to reflect this change.
|
||||
|
||||
*`var.inputs`*::
|
||||
+
|
||||
--
|
||||
* Value type is <<string,string>>
|
||||
* Default value is “eventbroker”
|
||||
--
|
||||
+
|
||||
Set the input(s) to expose for the Logstash ArcSight module. Valid settings are
|
||||
“eventbroker”, “smartconnector”, or “eventbroker,smartconnector” (exposes both
|
||||
inputs concurrently).
|
||||
|
||||
*`var.input.eventbroker.bootstrap_servers`*::
|
||||
+
|
||||
--
|
||||
* Value type is <<string,string>>
|
||||
* Default value is “localhost:39092”
|
||||
--
|
||||
+
|
||||
A list of EB URLs to use for establishing the initial connection to the cluster.
|
||||
This list should be in the form of `host1:port1,host2:port2`. These URLs are
|
||||
just used for the initial connection to discover the full cluster membership
|
||||
(which may change dynamically), so this list need not contain the full set of
|
||||
servers (you may want more than one, though, in case a server is down).
|
||||
|
||||
*`var.input.eventbroker.topics`*::
|
||||
+
|
||||
--
|
||||
* Value type is <<array,array>>
|
||||
* Default value is [“eb-cef”]
|
||||
--
|
||||
+
|
||||
A list of EB topics to subscribe to.
|
||||
|
||||
*`var.input.smartconnector.port`*::
|
||||
+
|
||||
--
|
||||
* Value type is <<number,number>>
|
||||
* Default value is 5000
|
||||
--
|
||||
+
|
||||
The TCP port to listen on when receiving data from SCs.
|
||||
|
||||
*`var.elasticsearch.hosts`*::
|
||||
+
|
||||
--
|
||||
* Value type is <<uri,uri>>
|
||||
* Default value is “localhost:9200”
|
||||
--
|
||||
+
|
||||
Sets the host(s) of the Elasticsearch cluster. If given an <<array,array>> it
|
||||
will load balance requests across the hosts specified in the hosts parameter. It
|
||||
is important to exclude {ref}/modules-node.html[dedicated master nodes] from the
|
||||
hosts list to prevent Logstash from sending bulk requests to the master nodes.
|
||||
So this parameter should only reference either data or client nodes in
|
||||
Elasticsearch.
|
||||
+
|
||||
Any special characters present in the URLs here MUST be URL escaped! This means #
|
||||
should be put in as %23 for instance.
|
||||
|
||||
*`var.elasticsearch.username`*::
|
||||
+
|
||||
--
|
||||
* Value type is <<string,string>>
|
||||
* Default value is “elastic”
|
||||
--
|
||||
+
|
||||
The username to authenticate to a secure Elasticsearch cluster.
|
||||
|
||||
*`var.elasticsearch.password`*::
|
||||
+
|
||||
--
|
||||
* Value type is <<string,string>>
|
||||
* Default value is “changeme”
|
||||
--
|
||||
+
|
||||
The password to authenticate to a secure Elasticsearch cluster.
|
||||
|
||||
*`var.elasticsearch.ssl.enabled`*::
|
||||
+
|
||||
--
|
||||
* Value type is <<boolean,boolean>>
|
||||
* There is no default value for this setting.
|
||||
--
|
||||
+
|
||||
Enable SSL/TLS secured communication to the Elasticsearch cluster. Leaving this
|
||||
unspecified will use whatever scheme is specified in the URLs listed in `hosts`.
|
||||
If no explicit protocol is specified, plain HTTP will be used. If SSL is
|
||||
explicitly disabled here, the plugin will refuse to start if an HTTPS URL is
|
||||
given in hosts.
|
||||
|
||||
*`var.elasticsearch.ssl.verification_mode`*::
|
||||
+
|
||||
--
|
||||
* Value type is <<string,string>>
|
||||
* Default value is "strict"
|
||||
--
|
||||
+
|
||||
The hostname verification setting when communicating with Elasticsearch. Set to
|
||||
`disable` to turn off hostname verification. Disabling this has serious security
|
||||
concerns.
|
||||
|
||||
*`var.elasticsearch.ssl.certificate_authority`*::
|
||||
+
|
||||
--
|
||||
* Value type is <<string,string>>
|
||||
* There is no default value for this setting
|
||||
--
|
||||
+
|
||||
The path to an X.509 certificate to use to validate SSL certificates when
|
||||
communicating with Kibana.
|
||||
|
||||
*`var.elasticsearch.ssl.certificate`*::
|
||||
+
|
||||
--
|
||||
* Value type is <<string,string>>
|
||||
* There is no default value for this setting
|
||||
--
|
||||
+
|
||||
The path to an X.509 certificate to use for client authentication when
|
||||
communicating with Elasticsearch.
|
||||
|
||||
*`var.elasticsearch.ssl.key`*::
|
||||
+
|
||||
--
|
||||
* Value type is <<string,string>>
|
||||
* There is no default value for this setting
|
||||
--
|
||||
+
|
||||
The path to the certificate key for client authentication when communicating
|
||||
with Elasticsearch.
|
||||
|
||||
*`var.kibana.host`*::
|
||||
+
|
||||
--
|
||||
* Value type is <<string,string>>
|
||||
* Default value is “localhost:5601”
|
||||
--
|
||||
+
|
||||
Sets the host of the Kibana instance to import dashboards and visualizations.
|
||||
|
||||
*`var.kibana.username`*::
|
||||
+
|
||||
--
|
||||
* Value type is <<string,string>>
|
||||
* Default value is “elastic”
|
||||
--
|
||||
+
|
||||
The username to authenticate to a secured Kibana instance.
|
||||
|
||||
*`var.kibana.password`*::
|
||||
+
|
||||
--
|
||||
* Value type is <<string,string>>
|
||||
* Default value is “changeme”
|
||||
--
|
||||
+
|
||||
The password to authenticate to a secure Kibana instance.
|
||||
|
||||
*`var.kibana.ssl.enabled`*::
|
||||
+
|
||||
--
|
||||
* Value type is <<boolean,boolean>>
|
||||
* Default value is false
|
||||
--
|
||||
+
|
||||
Enable SSL/TLS secured communication to the Kibana instance.
|
||||
|
||||
*`var.kibana.ssl.verification_mode`*::
|
||||
+
|
||||
--
|
||||
* Value type is <<string,string>>
|
||||
* Default value is "strict"
|
||||
--
|
||||
+
|
||||
The hostname verification setting when communicating with Kibana. Set to
|
||||
`disable` to turn off hostname verification. Disabling this has serious security
|
||||
concerns.
|
||||
|
||||
*`var.kibana.ssl.certificate_authority`*::
|
||||
+
|
||||
--
|
||||
* Value type is <<string,string>>
|
||||
* There is no default value for this setting
|
||||
--
|
||||
+
|
||||
The path to an X.509 certificate to use to validate SSL certificates when
|
||||
communicating with Kibana.
|
||||
|
||||
*`var.kibana.ssl.certificate`*::
|
||||
+
|
||||
--
|
||||
* Value type is <<string,string>>
|
||||
* There is no default value for this setting
|
||||
--
|
||||
+
|
||||
The path to an X.509 certificate to use for client authentication when
|
||||
communicating with Kibana.
|
||||
|
||||
*`var.kibana.ssl.key`*::
|
||||
+
|
||||
--
|
||||
* Value type is <<string,string>>
|
||||
* There is no default value for this setting
|
||||
--
|
||||
+
|
||||
The path to the certificate key for client authentication when communicating
|
||||
with Kibana.
|
||||
Loading…
Add table
Add a link
Reference in a new issue