mirror of
https://github.com/elastic/logstash.git
synced 2025-04-24 22:57:16 -04:00
6832040d28
This work breaks out the JVM setting info into a new section, and expands and updates the content. It adds new subheadings to make scanning the content easier.
334 lines
11 KiB
Text
334 lines
11 KiB
Text
[[getting-started-with-logstash]]
|
||
== Getting Started with Logstash
|
||
|
||
This section guides you through the process of installing Logstash and verifying that everything is running properly.
|
||
After learning how to stash your first event, you go on to create a more advanced pipeline that takes Apache web logs as
|
||
input, parses the logs, and writes the parsed data to an Elasticsearch cluster. Then you learn how to stitch together multiple input and output plugins to unify data from a variety of disparate sources.
|
||
|
||
This section includes the following topics:
|
||
|
||
* <<ls-jvm>>
|
||
* <<installing-logstash>>
|
||
* <<first-event>>
|
||
* {logstash-ref}/advanced-pipeline.html[Parsing Logs with Logstash]
|
||
* {logstash-ref}/multiple-input-output-plugins.html[Stitching Together Multiple Input and Output Plugins]
|
||
|
||
include::jvm.asciidoc[]
|
||
|
||
[[installing-logstash]]
|
||
=== Installing Logstash
|
||
|
||
[float]
|
||
[[installing-binary]]
|
||
=== Installing from a Downloaded Binary
|
||
|
||
The {ls} binaries are available from
|
||
https://www.elastic.co/downloads/logstash[https://www.elastic.co/downloads].
|
||
Download the Logstash installation file for your host environment--TARG.GZ, DEB,
|
||
ZIP, or RPM.
|
||
|
||
Unpack the file. Do not install Logstash into a directory path that
|
||
contains colon (:) characters.
|
||
|
||
[NOTE]
|
||
--
|
||
These packages are free to use under the Elastic license. They contain open
|
||
source and free commercial features and access to paid commercial features.
|
||
{stack-ov}/license-management.html[Start a 30-day trial] to try out all of the
|
||
paid commercial features. See the
|
||
https://www.elastic.co/subscriptions[Subscriptions] page for information about
|
||
Elastic license levels.
|
||
|
||
Alternatively, you can download an `oss` package, which contains only features
|
||
that are available under the Apache 2.0 license.
|
||
--
|
||
|
||
On supported Linux operating systems, you can use a package manager to install Logstash.
|
||
|
||
[float]
|
||
[[package-repositories]]
|
||
=== Installing from Package Repositories
|
||
|
||
We also have repositories available for APT and YUM based distributions. Note
|
||
that we only provide binary packages, but no source packages, as the packages
|
||
are created as part of the Logstash build.
|
||
|
||
We have split the Logstash package repositories by version into separate urls
|
||
to avoid accidental upgrades across major versions. For all {major-version}.y
|
||
releases use {major-version} as version number.
|
||
|
||
We use the PGP key
|
||
https://pgp.mit.edu/pks/lookup?op=vindex&search=0xD27D666CD88E42B4[D88E42B4],
|
||
Elastic's Signing Key, with fingerprint
|
||
|
||
4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4
|
||
|
||
to sign all our packages. It is available from https://pgp.mit.edu.
|
||
|
||
[float]
|
||
==== APT
|
||
|
||
ifeval::["{release-state}"=="unreleased"]
|
||
|
||
Version {logstash_version} of Logstash has not yet been released.
|
||
|
||
endif::[]
|
||
|
||
ifeval::["{release-state}"!="unreleased"]
|
||
|
||
Download and install the Public Signing Key:
|
||
|
||
[source,sh]
|
||
--------------------------------------------------
|
||
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
|
||
--------------------------------------------------
|
||
|
||
You may need to install the `apt-transport-https` package on Debian before proceeding:
|
||
|
||
[source,sh]
|
||
--------------------------------------------------
|
||
sudo apt-get install apt-transport-https
|
||
--------------------------------------------------
|
||
|
||
// THIS IS A NESTED STATEMENT - This block executes if release-state != unreleased and release-state == released
|
||
|
||
ifeval::["{release-state}"=="released"]
|
||
|
||
Save the repository definition to +/etc/apt/sources.list.d/elastic-{major-version}.list+:
|
||
|
||
["source","sh",subs="attributes"]
|
||
--------------------------------------------------
|
||
echo "deb https://artifacts.elastic.co/packages/{major-version}/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-{major-version}.list
|
||
--------------------------------------------------
|
||
|
||
endif::[]
|
||
|
||
// THIS IS A NESTED STATEMENT - This block executes if release-state != unreleased and release-state == prerelase
|
||
|
||
ifeval::["{release-state}"=="prerelease"]
|
||
|
||
Save the repository definition to +/etc/apt/sources.list.d/elastic-{major-version}-prerelease.list+:
|
||
|
||
["source","sh",subs="attributes"]
|
||
--------------------------------------------------
|
||
echo "deb https://artifacts.elastic.co/packages/{major-version}-prerelease/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-{major-version}-prerelease.list
|
||
--------------------------------------------------
|
||
|
||
endif::[]
|
||
|
||
[WARNING]
|
||
==================================================
|
||
Use the `echo` method described above to add the Logstash repository. Do not
|
||
use `add-apt-repository` as it will add a `deb-src` entry as well, but we do not
|
||
provide a source package. If you have added the `deb-src` entry, you will see an
|
||
error like the following:
|
||
|
||
Unable to find expected entry 'main/source/Sources' in Release file (Wrong sources.list entry or malformed file)
|
||
|
||
Just delete the `deb-src` entry from the `/etc/apt/sources.list` file and the
|
||
installation should work as expected.
|
||
==================================================
|
||
|
||
Run `sudo apt-get update` and the repository is ready for use. You can install
|
||
it with:
|
||
|
||
[source,sh]
|
||
--------------------------------------------------
|
||
sudo apt-get update && sudo apt-get install logstash
|
||
--------------------------------------------------
|
||
|
||
See {logstash-ref}/running-logstash.html[Running Logstash] for details about managing Logstash as a system service.
|
||
|
||
endif::[]
|
||
|
||
[float]
|
||
==== YUM
|
||
|
||
ifeval::["{release-state}"=="unreleased"]
|
||
|
||
Version {logstash_version} of Logstash has not yet been released.
|
||
|
||
endif::[]
|
||
|
||
ifeval::["{release-state}"!="unreleased"]
|
||
|
||
Download and install the public signing key:
|
||
|
||
[source,sh]
|
||
--------------------------------------------------
|
||
sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
|
||
--------------------------------------------------
|
||
|
||
Add the following in your `/etc/yum.repos.d/` directory
|
||
in a file with a `.repo` suffix, for example `logstash.repo`
|
||
|
||
// THIS IS A NESTED STATEMENT - This block executes if release-state != unreleased and release-state == prerelase
|
||
|
||
ifeval::["{release-state}"=="prerelease"]
|
||
|
||
["source","sh",subs="attributes"]
|
||
--------------------------------------------------
|
||
[logstash-{major-version}]
|
||
name=Elastic repository for {major-version} packages
|
||
baseurl=https://artifacts.elastic.co/packages/{major-version}-prerelease/yum
|
||
gpgcheck=1
|
||
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
|
||
enabled=1
|
||
autorefresh=1
|
||
type=rpm-md
|
||
--------------------------------------------------
|
||
|
||
endif::[]
|
||
|
||
// THIS IS A NESTED STATEMENT - This block executes if release-state != unreleased and release-state == released
|
||
|
||
ifeval::["{release-state}"=="released"]
|
||
|
||
["source","sh",subs="attributes"]
|
||
--------------------------------------------------
|
||
[logstash-{major-version}]
|
||
name=Elastic repository for {major-version} packages
|
||
baseurl=https://artifacts.elastic.co/packages/{major-version}/yum
|
||
gpgcheck=1
|
||
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
|
||
enabled=1
|
||
autorefresh=1
|
||
type=rpm-md
|
||
--------------------------------------------------
|
||
|
||
endif::[]
|
||
|
||
And your repository is ready for use. You can install it with:
|
||
|
||
[source,sh]
|
||
--------------------------------------------------
|
||
sudo yum install logstash
|
||
--------------------------------------------------
|
||
|
||
WARNING: The repositories do not work with older rpm based distributions
|
||
that still use RPM v3, like CentOS5.
|
||
|
||
See the {logstash-ref}/running-logstash.html[Running Logstash] document for managing Logstash as a system service.
|
||
|
||
endif::[]
|
||
|
||
[float]
|
||
[[brew]]
|
||
=== Installing {ls} on macOS with Homebrew
|
||
|
||
Elastic publishes Homebrew formulae so you can install {ls} with the
|
||
https://brew.sh/[Homebrew] package manager.
|
||
|
||
To install with Homebrew, you first need to tap the Elastic Homebrew repository:
|
||
|
||
[source,sh]
|
||
-------------------------
|
||
brew tap elastic/tap
|
||
-------------------------
|
||
|
||
After you've tapped the Elastic Homebrew repo, you can use `brew install` to
|
||
install the default distribution of {ls}:
|
||
|
||
[source,sh]
|
||
-------------------------
|
||
brew install elastic/tap/logstash-full
|
||
-------------------------
|
||
|
||
This installs the most recently released default distribution of {ls}.
|
||
To install the OSS distribution, specify `elastic/tap/logstash-oss`.
|
||
|
||
|
||
[float]
|
||
[[brew-start]]
|
||
==== Starting {ls} with Homebrew
|
||
|
||
To have launchd start elastic/tap/logstash-full now and restart at login, run:
|
||
|
||
[source,sh]
|
||
-----
|
||
brew services start elastic/tap/logstash-full
|
||
-----
|
||
|
||
To run {ls}, in the foreground, run:
|
||
|
||
[source,sh]
|
||
-----
|
||
logstash
|
||
-----
|
||
|
||
|
||
==== Docker
|
||
|
||
Images are available for running Logstash as a Docker container. They are
|
||
available from the Elastic Docker registry.
|
||
|
||
See <<docker,Running Logstash on Docker>> for
|
||
details on how to configure and run Logstash Docker containers.
|
||
|
||
[[first-event]]
|
||
=== Stashing Your First Event
|
||
|
||
First, let's test your Logstash installation by running the most basic _Logstash pipeline_.
|
||
|
||
A Logstash pipeline has two required elements, `input` and `output`, and one optional element, `filter`. The input
|
||
plugins consume data from a source, the filter plugins modify the data as you specify, and the output plugins write
|
||
the data to a destination.
|
||
|
||
//TODO: REPLACE WITH NEW IMAGE
|
||
|
||
image::static/images/basic_logstash_pipeline.png[]
|
||
|
||
To test your Logstash installation, run the most basic Logstash pipeline. For
|
||
example:
|
||
|
||
["source","sh",subs="attributes"]
|
||
--------------------------------------------------
|
||
cd logstash-{logstash_version}
|
||
bin/logstash -e 'input { stdin { } } output { stdout {} }'
|
||
--------------------------------------------------
|
||
|
||
NOTE: The location of the `bin` directory varies by platform. See {logstash-ref}/dir-layout.html[Directory layout]
|
||
to find the location of `bin\logstash` on your system.
|
||
|
||
[IMPORTANT]
|
||
.macOS Gatekeeper warnings
|
||
====
|
||
Apple's rollout of stricter notarization requirements affected the notarization
|
||
of the {version} {ls} artifacts. If macOS Catalina displays a dialog when you
|
||
first run {ls} that interrupts it, you will need to take an action to allow it
|
||
to run.
|
||
To prevent Gatekeeper checks on the {ls} files, run the following command on the
|
||
downloaded `.tar.gz` archive or the directory to which was extracted:
|
||
[source,sh]
|
||
----
|
||
xattr -d -r com.apple.quarantine <archive-or-directory>
|
||
----
|
||
For example, if the `.tar.gz` file was extracted to the default
|
||
logstash-{version} directory, the command is:
|
||
[source,sh,subs="attributes"]
|
||
----
|
||
xattr -d -r com.apple.quarantine logstash-{version}
|
||
----
|
||
Alternatively, you can add a security override if a Gatekeeper popup appears by
|
||
following the instructions in the _How to open an app that hasn’t been notarized
|
||
or is from an unidentified developer_ section of
|
||
https://support.apple.com/en-us/HT202491[Safely open apps on your Mac].
|
||
====
|
||
|
||
|
||
|
||
The `-e` flag enables you to specify a configuration directly from the command line. Specifying configurations at the
|
||
command line lets you quickly test configurations without having to edit a file between iterations.
|
||
The pipeline in the example takes input from the standard input, `stdin`, and moves that input to the standard output,
|
||
`stdout`, in a structured format.
|
||
|
||
After starting Logstash, wait until you see "Pipeline main started" and then enter `hello world` at the command prompt:
|
||
|
||
[source,shell]
|
||
hello world
|
||
2013-11-21T01:22:14.405+0000 0.0.0.0 hello world
|
||
|
||
Logstash adds timestamp and IP address information to the message. Exit Logstash by issuing a *CTRL-D* command in the
|
||
shell where Logstash is running.
|
||
|
||
Congratulations! You've created and run a basic Logstash pipeline. Next, you learn how to create a more realistic pipeline.
|