mirror of
https://github.com/elastic/logstash.git
synced 2025-06-27 17:08:55 -04:00
90 lines
2.7 KiB
Text
90 lines
2.7 KiB
Text
[[ingest-converter]]
|
|
=== Converting Ingest Node Pipelines
|
|
|
|
After implementing {ref}/ingest.html[ingest] pipelines to parse your data, you
|
|
might decide that you want to take advantage of the richer transformation
|
|
capabilities in Logstash. For example, you may need to use Logstash instead of
|
|
ingest pipelines if you want to:
|
|
|
|
* Ingest from more inputs. Logstash can natively ingest data from many other
|
|
sources like TCP, UDP, syslog, and relational databases.
|
|
|
|
* Use multiple outputs. Ingest node was designed to only support Elasticsearch
|
|
as an output, but you may want to use more than one output. For example, you may
|
|
want to archive your incoming data to S3 as well as indexing it in
|
|
Elasticsearch.
|
|
|
|
* Take advantage of the richer transformation capabilities in Logstash, such as
|
|
external lookups.
|
|
|
|
* Use the persistent queue feature to handle spikes when ingesting data (from
|
|
Beats and other sources).
|
|
|
|
To make it easier for you to migrate your configurations, Logstash provides an
|
|
ingest pipeline conversion tool. The conversion tool takes the ingest pipeline
|
|
definition as input and, when possible, creates the equivalent Logstash
|
|
configuration as output.
|
|
|
|
See <<ingest-converter-limitations>> for a full list of tool limitations.
|
|
|
|
[[ingest-converter-run]]
|
|
==== Running the tool
|
|
|
|
You'll find the conversion tool in the `bin` directory of your Logstash
|
|
installation. See <<dir-layout>> to find the location of `bin` on your system.
|
|
|
|
To run the conversion tool, use the following command:
|
|
|
|
[source,shell]
|
|
-----
|
|
bin/ingest-convert.sh --input INPUT_FILE_URI --output OUTPUT_FILE_URI [--append-stdio]
|
|
-----
|
|
|
|
Where:
|
|
|
|
* `INPUT_FILE_URI` is a file URI that specifies the full path to the JSON file
|
|
that defines the ingest node pipeline.
|
|
|
|
* `OUTPUT_FILE_URI` is the file URI of the Logstash DSL file that will be
|
|
generated by the tool.
|
|
|
|
* `--append-stdio` is an optional flag that adds stdin and stdout sections to
|
|
the config instead of adding the default Elasticsearch output.
|
|
|
|
This command expects a file URI, so make sure you use forward slashes and
|
|
specify the full path to the file.
|
|
|
|
For example:
|
|
|
|
[source,text]
|
|
-----
|
|
bin/ingest-convert.sh --input file:///tmp/ingest/apache.json --output file:///tmp/ingest/apache.conf
|
|
-----
|
|
|
|
|
|
[[ingest-converter-limitations]]
|
|
==== Limitations
|
|
|
|
* Painless script conversion is not supported.
|
|
|
|
* Only a subset of available processors are
|
|
<<ingest-converter-supported-processors,supported>> for conversion. For
|
|
processors that are not supported, the tool produces a warning and continues
|
|
with a best-effort conversion.
|
|
|
|
[[ingest-converter-supported-processors]]
|
|
==== Supported Processors
|
|
|
|
The following ingest node processors are currently supported for conversion by
|
|
the tool:
|
|
|
|
* Append
|
|
* Convert
|
|
* Date
|
|
* GeoIP
|
|
* Grok
|
|
* Gsub
|
|
* Json
|
|
* Lowercase
|
|
* Rename
|
|
* Set
|