[[setup-logstash]]
== Setting Up and Running Logstash

Before reading this section, see <<installing-logstash>> for basic installation instructions to get you started.

This section includes additional information on how to set up and run Logstash, including:

* <<dir-layout>>
* <<config-setting-files>>
* <<running-logstash>>
* <<docker>>
* <<logstash-settings-file>>
* <<command-line-flags>>
* <<logging>>
* <<persistent-queues>>
* <<shutdown>>


[[dir-layout]]
=== Logstash Directory Layout

This section describes the default directory structure that is created when you unpack the Logstash installation packages.

[[zip-targz-layout]]
==== Directory Layout of `.zip` and `.tar.gz` Archives

The `.zip` and `.tar.gz` packages are entirely self-contained. All files and
directories are, by default, contained within the home directory -- the directory
created when unpacking the archive.

This is very convenient because you don't have to create any directories to start using Logstash, and uninstalling
Logstash is as easy as removing the home directory.  However, it is advisable to change the default locations of the
config and the logs directories so that you do not delete important data later on.

[cols="<h,<,<m,<m",options="header",]
|=======================================================================
| Type | Description | Default Location | Setting
| home
  | Home directory of the Logstash installation.
  | `{extract.path}`- Directory created by unpacking the archive
 d|

| bin
  | Binary scripts, including `logstash` to start Logstash
    and `logstash-plugin` to install plugins
  | `{extract.path}/bin`
 d|

| settings
  | Configuration files, including `logstash.yml` and `jvm.options`
  | `{extract.path}/config`
  | `path.settings`
  
| logs
  | Log files
  | `{extract.path}/logs`
  | `path.logs`

| plugins
  | Local, non Ruby-Gem plugin files. Each plugin is contained in a subdirectory. Recommended for development only.
  | `{extract.path}/plugins`
  | `path.plugins`

|=======================================================================

[[deb-layout]]
==== Directory Layout of Debian and RPM Packages

The Debian package and the RPM package each place config files, logs, and the settings files in the appropriate
locations for the system:

[cols="<h,<,<m,<m",options="header",]
|=======================================================================
| Type | Description | Default Location | Setting
| home
  | Home directory of the Logstash installation.
  | `/usr/share/logstash`
 d|

| bin
  | Binary scripts including `logstash` to start Logstash
    and `logstash-plugin` to install plugins
  | `/usr/share/logstash/bin`
 d|

| settings
  | Configuration files, including `logstash.yml`, `jvm.options`, and `startup.options`
  | `/etc/logstash`
  | `path.settings`

| conf
  | Logstash pipeline configuration files
  | `/etc/logstash/conf.d`
  | `path.config`

| logs
  | Log files
  | `/var/log/logstash`
  | `path.logs`

| plugins
  | Local, non Ruby-Gem plugin files. Each plugin is contained in a subdirectory. Recommended for development only.
  | `/usr/share/logstash/plugins`
  | `path.plugins`

|=======================================================================

[[docker-layout]]
==== Directory Layout of Docker Images

The Docker images are created from the `.tar.gz` packages, and follow a
similar directory layout.

[cols="<h,<,<m,<m",options="header",]
|=======================================================================
| Type | Description | Default Location | Setting
| home
  | Home directory of the Logstash installation.
  | `/usr/share/logstash`
 d|

| bin
  | Binary scripts, including `logstash` to start Logstash
    and `logstash-plugin` to install plugins
  | `/usr/share/logstash/bin`
 d|

| settings
  | Configuration files, including `logstash.yml` and `jvm.options`
  | `/usr/share/logstash/config`
  | `path.settings`

| conf
  | Logstash pipeline configuration files
  | `/usr/share/logstash/pipeline`
  | `path.config`

| plugins
  | Local, non Ruby-Gem plugin files. Each plugin is contained in a subdirectory. Recommended for development only.
  | `/usr/share/logstash/plugins`
  | `path.plugins`

|=======================================================================

NOTE: Logstash Docker containers do not create log files by default. They log
to standard output.

[[config-setting-files]]
=== Logstash Configuration Files

Logstash has two types of configuration files: _pipeline configuration files_, which define the Logstash processing
pipeline, and _settings files_, which specify options that control Logstash startup and execution.

==== Pipeline Configuration Files

You create pipeline configuration files when you define the stages of your Logstash processing pipeline. On deb and
rpm, you place the pipeline configuration files in the `/etc/logstash/conf.d` directory. Logstash tries to load all
files in the `/etc/logstash/conf.d directory`, so don't store any non-config files or backup files in this directory.

See <<configuration>> for more info.

==== Settings Files

The settings files are already defined in the Logstash installation. Logstash includes the following settings files:

*`logstash.yml`*::
  Contains Logstash configuration flags. You can set flags in this file instead of passing the flags at the command
  line. Any flags that you set at the command line override the corresponding settings in the `logstash.yml` file. See <<logstash-settings-file>> for more info.
*`jvm.options`*::
  Contains JVM configuration flags. Specify each flag on a separate line. You can also use this file to set the locale
  for Logstash.
*`startup.options` (Linux)*::
  Contains options used by the `system-install` script in `/usr/share/logstash/bin` to build the appropriate startup
  script for your system. When you install the Logstash package, the `system-install` script executes at the end of the
  installation process and uses the settings specified in `startup.options` to set options such as the user, group,
  service name, and service description. By default, Logstash services are installed under the user `logstash`. The `startup.options` file makes it easier for you to install multiple instances of the Logstash service. You can copy
  the file and change the values for specific settings. Note that the `startup.options` file is not read at startup. If
  you want to change the Logstash startup script (for example, to change the Logstash user or read from a different
  configuration path), you must re-run the `system-install` script (as root) to pass in the new settings.

[[running-logstash]]
=== Running Logstash as a Service on Debian or RPM

Logstash is not started automatically after installation. How to start and stop Logstash depends on whether your system
uses systemd, upstart, or SysV.

Here are some common operating systems and versions, and the corresponding
startup styles they use.  This list is intended to be informative, not exhaustive.

|=======================================================================
| Distribution | Service System |
| Ubuntu 16.04 and newer | <<running-logstash-systemd,systemd>> |
| Ubuntu 12.04 through 15.10 | <<running-logstash-upstart,upstart>> |
| Debian 8 "jessie" and newer | <<running-logstash-systemd,systemd>> |
| Debian 7 "wheezy" and older | <<running-logstash-sysv,sysv>> |
| CentOS (and RHEL) 7 and newer | <<running-logstash-systemd,systemd>> |
| CentOS (and RHEL) 6 | <<running-logstash-upstart,upstart>> |
|=======================================================================

[[running-logstash-systemd]]
==== Running Logstash by Using Systemd

Distributions like Debian Jessie, Ubuntu 15.10+, and many of the SUSE derivatives use systemd and the
`systemctl` command to start and stop services. Logstash places the systemd unit files in `/etc/systemd/system` for both deb and rpm. After installing the package, you can start up Logstash with:

[source,sh]
--------------------------------------------
sudo systemctl start logstash.service
-------------------------------------------

[[running-logstash-upstart]]
==== Running Logstash by Using Upstart

For systems that use upstart, you can start Logstash with:

[source,sh]
--------------------------------------------
sudo initctl start logstash
-------------------------------------------

The auto-generated configuration file for upstart systems is `/etc/init/logstash.conf`.

[[running-logstash-sysv]]
==== Running Logstash by Using SysV

For systems that use SysV, you can start Logstash with:

[source,sh]
--------------------------------------------
sudo /etc/init.d/logstash start
-------------------------------------------

The auto-generated configuration file for SysV systems is `/etc/init.d/logstash`.