mirror of
https://github.com/elastic/logstash.git
synced 2025-04-24 22:57:16 -04:00
305 lines
9.3 KiB
HTML
305 lines
9.3 KiB
HTML
---
|
|
title: logstash docs for inputs/zeromq
|
|
layout: content_right
|
|
---
|
|
<h2>zeromq</h2>
|
|
<h3>Milestone: <a href="../plugin-milestones">2</a></h3>
|
|
|
|
<p>Read events over a 0MQ SUB socket.</p>
|
|
|
|
<p>You need to have the 0mq 2.1.x library installed to be able to use
|
|
this input plugin.</p>
|
|
|
|
<p>The default settings will create a subscriber binding to tcp://127.0.0.1:2120
|
|
waiting for connecting publishers.</p>
|
|
|
|
|
|
<h3> Synopsis </h3>
|
|
|
|
This is what it might look like in your config file:
|
|
|
|
<pre><code>input {
|
|
zeromq {
|
|
<a href="#add_field">add_field</a> => ... # hash (optional), default: {}
|
|
<a href="#address">address</a> => ... # array (optional), default: ["tcp://*:2120"]
|
|
<a href="#codec">codec</a> => ... # codec (optional), default: "plain"
|
|
<a href="#debug">debug</a> => ... # boolean (optional), default: false
|
|
<a href="#mode">mode</a> => ... # string, one of ["server", "client"] (optional), default: "server"
|
|
<a href="#sender">sender</a> => ... # string (optional)
|
|
<a href="#sockopt">sockopt</a> => ... # hash (optional)
|
|
<a href="#tags">tags</a> => ... # array (optional)
|
|
<a href="#topic">topic</a> => ... # array (optional)
|
|
<a href="#topology">topology</a> => ... # string, one of ["pushpull", "pubsub", "pair"] (required)
|
|
<a href="#type">type</a> => ... # string (optional)
|
|
}
|
|
|
|
}
|
|
</code></pre>
|
|
|
|
<h3> Details </h3>
|
|
|
|
<h4>
|
|
<a name="add_field">
|
|
add_field
|
|
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> Value type is <a href="../configuration#hash">hash</a> </li>
|
|
<li> Default value is {} </li>
|
|
</ul>
|
|
|
|
<p>Add a field to an event</p>
|
|
|
|
<h4>
|
|
<a name="address">
|
|
address
|
|
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> Value type is <a href="../configuration#array">array</a> </li>
|
|
<li> Default value is ["tcp://*:2120"] </li>
|
|
</ul>
|
|
|
|
<p>0mq socket address to connect or bind
|
|
Please note that <code>inproc://</code> will not work with logstash
|
|
as each we use a context per thread.
|
|
By default, inputs bind/listen
|
|
and outputs connect</p>
|
|
|
|
<h4>
|
|
<a name="charset">
|
|
charset
|
|
<strong>DEPRECATED</strong>
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> DEPRECATED WARNING: This config item is deprecated. It may be removed in a further version. </li>
|
|
<li> Value can be any of: "ASCII-8BIT", "UTF-8", "US-ASCII", "Big5", "Big5-HKSCS", "Big5-UAO", "CP949", "Emacs-Mule", "EUC-JP", "EUC-KR", "EUC-TW", "GB18030", "GBK", "ISO-8859-1", "ISO-8859-2", "ISO-8859-3", "ISO-8859-4", "ISO-8859-5", "ISO-8859-6", "ISO-8859-7", "ISO-8859-8", "ISO-8859-9", "ISO-8859-10", "ISO-8859-11", "ISO-8859-13", "ISO-8859-14", "ISO-8859-15", "ISO-8859-16", "KOI8-R", "KOI8-U", "Shift_JIS", "UTF-16BE", "UTF-16LE", "UTF-32BE", "UTF-32LE", "Windows-1251", "BINARY", "IBM437", "CP437", "IBM737", "CP737", "IBM775", "CP775", "CP850", "IBM850", "IBM852", "CP852", "IBM855", "CP855", "IBM857", "CP857", "IBM860", "CP860", "IBM861", "CP861", "IBM862", "CP862", "IBM863", "CP863", "IBM864", "CP864", "IBM865", "CP865", "IBM866", "CP866", "IBM869", "CP869", "Windows-1258", "CP1258", "GB1988", "macCentEuro", "macCroatian", "macCyrillic", "macGreek", "macIceland", "macRoman", "macRomania", "macThai", "macTurkish", "macUkraine", "CP950", "Big5-HKSCS:2008", "CP951", "stateless-ISO-2022-JP", "eucJP", "eucJP-ms", "euc-jp-ms", "CP51932", "eucKR", "eucTW", "GB2312", "EUC-CN", "eucCN", "GB12345", "CP936", "ISO-2022-JP", "ISO2022-JP", "ISO-2022-JP-2", "ISO2022-JP2", "CP50220", "CP50221", "ISO8859-1", "Windows-1252", "CP1252", "ISO8859-2", "Windows-1250", "CP1250", "ISO8859-3", "ISO8859-4", "ISO8859-5", "ISO8859-6", "Windows-1256", "CP1256", "ISO8859-7", "Windows-1253", "CP1253", "ISO8859-8", "Windows-1255", "CP1255", "ISO8859-9", "Windows-1254", "CP1254", "ISO8859-10", "ISO8859-11", "TIS-620", "Windows-874", "CP874", "ISO8859-13", "Windows-1257", "CP1257", "ISO8859-14", "ISO8859-15", "ISO8859-16", "CP878", "Windows-31J", "CP932", "csWindows31J", "SJIS", "PCK", "MacJapanese", "MacJapan", "ASCII", "ANSI_X3.4-1968", "646", "UTF-7", "CP65000", "CP65001", "UTF8-MAC", "UTF-8-MAC", "UTF-8-HFS", "UTF-16", "UTF-32", "UCS-2BE", "UCS-4BE", "UCS-4LE", "CP1251", "UTF8-DoCoMo", "SJIS-DoCoMo", "UTF8-KDDI", "SJIS-KDDI", "ISO-2022-JP-KDDI", "stateless-ISO-2022-JP-KDDI", "UTF8-SoftBank", "SJIS-SoftBank", "locale", "external", "filesystem", "internal" </li>
|
|
<li> There is no default value for this setting. </li>
|
|
</ul>
|
|
|
|
<p>The character encoding used in this input. Examples include "UTF-8"
|
|
and "cp1252"</p>
|
|
|
|
<p>This setting is useful if your log files are in Latin-1 (aka cp1252)
|
|
or in another character set other than UTF-8.</p>
|
|
|
|
<p>This only affects "plain" format logs since json is UTF-8 already.</p>
|
|
|
|
<h4>
|
|
<a name="codec">
|
|
codec
|
|
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> Value type is <a href="../configuration#codec">codec</a> </li>
|
|
<li> Default value is "plain" </li>
|
|
</ul>
|
|
|
|
<p>The codec used for input data</p>
|
|
|
|
<h4>
|
|
<a name="debug">
|
|
debug
|
|
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> Value type is <a href="../configuration#boolean">boolean</a> </li>
|
|
<li> Default value is false </li>
|
|
</ul>
|
|
|
|
<p>Set this to true to enable debugging on an input.</p>
|
|
|
|
<h4>
|
|
<a name="format">
|
|
format
|
|
<strong>DEPRECATED</strong>
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> DEPRECATED WARNING: This config item is deprecated. It may be removed in a further version. </li>
|
|
<li> Value can be any of: "plain", "json", "json_event", "msgpack_event" </li>
|
|
<li> There is no default value for this setting. </li>
|
|
</ul>
|
|
|
|
<p>The format of input data (plain, json, json_event)</p>
|
|
|
|
<h4>
|
|
<a name="message_format">
|
|
message_format
|
|
<strong>DEPRECATED</strong>
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> DEPRECATED WARNING: This config item is deprecated. It may be removed in a further version. </li>
|
|
<li> Value type is <a href="../configuration#string">string</a> </li>
|
|
<li> There is no default value for this setting. </li>
|
|
</ul>
|
|
|
|
<p>If format is "json", an event sprintf string to build what
|
|
the display @message should be given (defaults to the raw JSON).
|
|
sprintf format strings look like %{fieldname}</p>
|
|
|
|
<p>If format is "json_event", ALL fields except for @type
|
|
are expected to be present. Not receiving all fields
|
|
will cause unexpected results.</p>
|
|
|
|
<h4>
|
|
<a name="mode">
|
|
mode
|
|
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> Value can be any of: "server", "client" </li>
|
|
<li> Default value is "server" </li>
|
|
</ul>
|
|
|
|
<p>mode
|
|
server mode binds/listens
|
|
client mode connects</p>
|
|
|
|
<h4>
|
|
<a name="sender">
|
|
sender
|
|
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> Value type is <a href="../configuration#string">string</a> </li>
|
|
<li> There is no default value for this setting. </li>
|
|
</ul>
|
|
|
|
<p>sender
|
|
overrides the sender to
|
|
set the source of the event
|
|
default is "zmq+topology://type/"</p>
|
|
|
|
<h4>
|
|
<a name="sockopt">
|
|
sockopt
|
|
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> Value type is <a href="../configuration#hash">hash</a> </li>
|
|
<li> There is no default value for this setting. </li>
|
|
</ul>
|
|
|
|
<p>0mq socket options
|
|
This exposes zmq_setsockopt
|
|
for advanced tuning
|
|
see http://api.zeromq.org/2-1:zmq-setsockopt for details</p>
|
|
|
|
<p>This is where you would set values like:
|
|
ZMQ::HWM - high water mark
|
|
ZMQ::IDENTITY - named queues
|
|
ZMQ::SWAP_SIZE - space for disk overflow</p>
|
|
|
|
<p>example: sockopt => ["ZMQ::HWM", 50, "ZMQ::IDENTITY", "my<em>named</em>queue"]</p>
|
|
|
|
<h4>
|
|
<a name="tags">
|
|
tags
|
|
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> Value type is <a href="../configuration#array">array</a> </li>
|
|
<li> There is no default value for this setting. </li>
|
|
</ul>
|
|
|
|
<p>Add any number of arbitrary tags to your event.</p>
|
|
|
|
<p>This can help with processing later.</p>
|
|
|
|
<h4>
|
|
<a name="topic">
|
|
topic
|
|
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> Value type is <a href="../configuration#array">array</a> </li>
|
|
<li> There is no default value for this setting. </li>
|
|
</ul>
|
|
|
|
<p>0mq topic
|
|
This is used for the 'pubsub' topology only
|
|
On inputs, this allows you to filter messages by topic
|
|
On outputs, this allows you to tag a message for routing
|
|
NOTE: ZeroMQ does subscriber side filtering.
|
|
NOTE: All topics have an implicit wildcard at the end
|
|
You can specify multiple topics here</p>
|
|
|
|
<h4>
|
|
<a name="topology">
|
|
topology (required setting)
|
|
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> Value can be any of: "pushpull", "pubsub", "pair" </li>
|
|
<li> There is no default value for this setting. </li>
|
|
</ul>
|
|
|
|
<p>0mq topology
|
|
The default logstash topologies work as follows:
|
|
* pushpull - inputs are pull, outputs are push
|
|
* pubsub - inputs are subscribers, outputs are publishers
|
|
* pair - inputs are clients, inputs are servers</p>
|
|
|
|
<p>If the predefined topology flows don't work for you,
|
|
you can change the 'mode' setting
|
|
TODO (lusis) add req/rep MAYBE
|
|
TODO (lusis) add router/dealer</p>
|
|
|
|
<h4>
|
|
<a name="type">
|
|
type
|
|
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> Value type is <a href="../configuration#string">string</a> </li>
|
|
<li> There is no default value for this setting. </li>
|
|
</ul>
|
|
|
|
<p>Add a 'type' field to all events handled by this input.</p>
|
|
|
|
<p>Types are used mainly for filter activation.</p>
|
|
|
|
<p>If you create an input with type "foobar", then only filters
|
|
which also have type "foobar" will act on them.</p>
|
|
|
|
<p>The type is also stored as part of the event itself, so you
|
|
can also use the type to search for in the web interface.</p>
|
|
|
|
<p>If you try to set a type on an event that already has one (for
|
|
example when you send an event from a shipper to an indexer) then
|
|
a new input will not override the existing type. A type set at
|
|
the shipper stays with that event for its life even
|
|
when sent to another LogStash server.</p>
|
|
|
|
|
|
<hr>
|
|
|
|
This is documentation from <a href="https://github.com/logstash/logstash/blob/v1.2.0.beta1/lib/logstash/inputs/zeromq.rb">lib/logstash/inputs/zeromq.rb</a>
|