mirror of
https://github.com/elastic/logstash.git
synced 2025-04-24 22:57:16 -04:00
311 lines
9.7 KiB
HTML
311 lines
9.7 KiB
HTML
---
|
|
title: logstash docs for inputs/file
|
|
layout: content_right
|
|
---
|
|
<h2>file</h2>
|
|
<h3>Milestone: <a href="../plugin-milestones">2</a></h3>
|
|
|
|
<p>Stream events from files.</p>
|
|
|
|
<p>By default, each event is assumed to be one line. If you
|
|
want to join lines, you'll want to use the multiline filter.</p>
|
|
|
|
<p>Files are followed in a manner similar to "tail -0F". File rotation
|
|
is detected and handled by this input.</p>
|
|
|
|
|
|
<h3> Synopsis </h3>
|
|
|
|
This is what it might look like in your config file:
|
|
|
|
<pre><code>input {
|
|
file {
|
|
<a href="#add_field">add_field</a> => ... # hash (optional), default: {}
|
|
<a href="#codec">codec</a> => ... # codec (optional), default: "plain"
|
|
<a href="#debug">debug</a> => ... # boolean (optional), default: false
|
|
<a href="#discover_interval">discover_interval</a> => ... # number (optional), default: 15
|
|
<a href="#exclude">exclude</a> => ... # array (optional)
|
|
<a href="#path">path</a> => ... # array (required)
|
|
<a href="#sincedb_path">sincedb_path</a> => ... # string (optional)
|
|
<a href="#sincedb_write_interval">sincedb_write_interval</a> => ... # number (optional), default: 15
|
|
<a href="#start_position">start_position</a> => ... # string, one of ["beginning", "end"] (optional), default: "end"
|
|
<a href="#stat_interval">stat_interval</a> => ... # number (optional), default: 1
|
|
<a href="#tags">tags</a> => ... # array (optional)
|
|
<a href="#type">type</a> => ... # string (optional)
|
|
}
|
|
|
|
}
|
|
</code></pre>
|
|
|
|
<h3> Details </h3>
|
|
|
|
<h4>
|
|
<a name="add_field">
|
|
add_field
|
|
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> Value type is <a href="../configuration#hash">hash</a> </li>
|
|
<li> Default value is {} </li>
|
|
</ul>
|
|
|
|
<p>Add a field to an event</p>
|
|
|
|
<h4>
|
|
<a name="charset">
|
|
charset
|
|
<strong>DEPRECATED</strong>
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> DEPRECATED WARNING: This config item is deprecated. It may be removed in a further version. </li>
|
|
<li> Value can be any of: "ASCII-8BIT", "UTF-8", "US-ASCII", "Big5", "Big5-HKSCS", "Big5-UAO", "CP949", "Emacs-Mule", "EUC-JP", "EUC-KR", "EUC-TW", "GB18030", "GBK", "ISO-8859-1", "ISO-8859-2", "ISO-8859-3", "ISO-8859-4", "ISO-8859-5", "ISO-8859-6", "ISO-8859-7", "ISO-8859-8", "ISO-8859-9", "ISO-8859-10", "ISO-8859-11", "ISO-8859-13", "ISO-8859-14", "ISO-8859-15", "ISO-8859-16", "KOI8-R", "KOI8-U", "Shift_JIS", "UTF-16BE", "UTF-16LE", "UTF-32BE", "UTF-32LE", "Windows-1251", "BINARY", "IBM437", "CP437", "IBM737", "CP737", "IBM775", "CP775", "CP850", "IBM850", "IBM852", "CP852", "IBM855", "CP855", "IBM857", "CP857", "IBM860", "CP860", "IBM861", "CP861", "IBM862", "CP862", "IBM863", "CP863", "IBM864", "CP864", "IBM865", "CP865", "IBM866", "CP866", "IBM869", "CP869", "Windows-1258", "CP1258", "GB1988", "macCentEuro", "macCroatian", "macCyrillic", "macGreek", "macIceland", "macRoman", "macRomania", "macThai", "macTurkish", "macUkraine", "CP950", "Big5-HKSCS:2008", "CP951", "stateless-ISO-2022-JP", "eucJP", "eucJP-ms", "euc-jp-ms", "CP51932", "eucKR", "eucTW", "GB2312", "EUC-CN", "eucCN", "GB12345", "CP936", "ISO-2022-JP", "ISO2022-JP", "ISO-2022-JP-2", "ISO2022-JP2", "CP50220", "CP50221", "ISO8859-1", "Windows-1252", "CP1252", "ISO8859-2", "Windows-1250", "CP1250", "ISO8859-3", "ISO8859-4", "ISO8859-5", "ISO8859-6", "Windows-1256", "CP1256", "ISO8859-7", "Windows-1253", "CP1253", "ISO8859-8", "Windows-1255", "CP1255", "ISO8859-9", "Windows-1254", "CP1254", "ISO8859-10", "ISO8859-11", "TIS-620", "Windows-874", "CP874", "ISO8859-13", "Windows-1257", "CP1257", "ISO8859-14", "ISO8859-15", "ISO8859-16", "CP878", "Windows-31J", "CP932", "csWindows31J", "SJIS", "PCK", "MacJapanese", "MacJapan", "ASCII", "ANSI_X3.4-1968", "646", "UTF-7", "CP65000", "CP65001", "UTF8-MAC", "UTF-8-MAC", "UTF-8-HFS", "UTF-16", "UTF-32", "UCS-2BE", "UCS-4BE", "UCS-4LE", "CP1251", "UTF8-DoCoMo", "SJIS-DoCoMo", "UTF8-KDDI", "SJIS-KDDI", "ISO-2022-JP-KDDI", "stateless-ISO-2022-JP-KDDI", "UTF8-SoftBank", "SJIS-SoftBank", "locale", "external", "filesystem", "internal" </li>
|
|
<li> There is no default value for this setting. </li>
|
|
</ul>
|
|
|
|
<p>The character encoding used in this input. Examples include "UTF-8"
|
|
and "cp1252"</p>
|
|
|
|
<p>This setting is useful if your log files are in Latin-1 (aka cp1252)
|
|
or in another character set other than UTF-8.</p>
|
|
|
|
<p>This only affects "plain" format logs since json is UTF-8 already.</p>
|
|
|
|
<h4>
|
|
<a name="codec">
|
|
codec
|
|
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> Value type is <a href="../configuration#codec">codec</a> </li>
|
|
<li> Default value is "plain" </li>
|
|
</ul>
|
|
|
|
<p>The codec used for input data</p>
|
|
|
|
<h4>
|
|
<a name="debug">
|
|
debug
|
|
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> Value type is <a href="../configuration#boolean">boolean</a> </li>
|
|
<li> Default value is false </li>
|
|
</ul>
|
|
|
|
<p>Set this to true to enable debugging on an input.</p>
|
|
|
|
<h4>
|
|
<a name="discover_interval">
|
|
discover_interval
|
|
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> Value type is <a href="../configuration#number">number</a> </li>
|
|
<li> Default value is 15 </li>
|
|
</ul>
|
|
|
|
<p>How often we expand globs to discover new files to watch.</p>
|
|
|
|
<h4>
|
|
<a name="exclude">
|
|
exclude
|
|
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> Value type is <a href="../configuration#array">array</a> </li>
|
|
<li> There is no default value for this setting. </li>
|
|
</ul>
|
|
|
|
<p>Exclusions (matched against the filename, not full path). Globs
|
|
are valid here, too. For example, if you have</p>
|
|
|
|
<pre><code>path => "/var/log/*"
|
|
</code></pre>
|
|
|
|
<p>you might want to exclude gzipped files:</p>
|
|
|
|
<pre><code>exclude => "*.gz"
|
|
</code></pre>
|
|
|
|
<h4>
|
|
<a name="format">
|
|
format
|
|
<strong>DEPRECATED</strong>
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> DEPRECATED WARNING: This config item is deprecated. It may be removed in a further version. </li>
|
|
<li> Value can be any of: "plain", "json", "json_event", "msgpack_event" </li>
|
|
<li> There is no default value for this setting. </li>
|
|
</ul>
|
|
|
|
<p>The format of input data (plain, json, json_event)</p>
|
|
|
|
<h4>
|
|
<a name="message_format">
|
|
message_format
|
|
<strong>DEPRECATED</strong>
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> DEPRECATED WARNING: This config item is deprecated. It may be removed in a further version. </li>
|
|
<li> Value type is <a href="../configuration#string">string</a> </li>
|
|
<li> There is no default value for this setting. </li>
|
|
</ul>
|
|
|
|
<p>If format is "json", an event sprintf string to build what
|
|
the display @message should be given (defaults to the raw JSON).
|
|
sprintf format strings look like %{fieldname}</p>
|
|
|
|
<p>If format is "json_event", ALL fields except for @type
|
|
are expected to be present. Not receiving all fields
|
|
will cause unexpected results.</p>
|
|
|
|
<h4>
|
|
<a name="path">
|
|
path (required setting)
|
|
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> Value type is <a href="../configuration#array">array</a> </li>
|
|
<li> There is no default value for this setting. </li>
|
|
</ul>
|
|
|
|
<p>TODO(sissel): This should switch to use the 'line' codec by default
|
|
once file following
|
|
The path to the file to use as an input.
|
|
You can use globs here, such as <code>/var/log/*.log</code>
|
|
Paths must be absolute and cannot be relative.</p>
|
|
|
|
<h4>
|
|
<a name="sincedb_path">
|
|
sincedb_path
|
|
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> Value type is <a href="../configuration#string">string</a> </li>
|
|
<li> There is no default value for this setting. </li>
|
|
</ul>
|
|
|
|
<p>Where to write the since database (keeps track of the current
|
|
position of monitored log files). The default will write
|
|
sincedb files to some path matching "$HOME/.sincedb*"</p>
|
|
|
|
<h4>
|
|
<a name="sincedb_write_interval">
|
|
sincedb_write_interval
|
|
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> Value type is <a href="../configuration#number">number</a> </li>
|
|
<li> Default value is 15 </li>
|
|
</ul>
|
|
|
|
<p>How often to write a since database with the current position of
|
|
monitored log files.</p>
|
|
|
|
<h4>
|
|
<a name="start_position">
|
|
start_position
|
|
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> Value can be any of: "beginning", "end" </li>
|
|
<li> Default value is "end" </li>
|
|
</ul>
|
|
|
|
<p>Choose where logstash starts initially reading files - at the beginning or
|
|
at the end. The default behavior treats files like live streams and thus
|
|
starts at the end. If you have old data you want to import, set this
|
|
to 'beginning'</p>
|
|
|
|
<p>This option only modifieds "first contact" situations where a file is new
|
|
and not seen before. If a file has already been seen before, this option
|
|
has no effect.</p>
|
|
|
|
<h4>
|
|
<a name="stat_interval">
|
|
stat_interval
|
|
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> Value type is <a href="../configuration#number">number</a> </li>
|
|
<li> Default value is 1 </li>
|
|
</ul>
|
|
|
|
<p>How often we stat files to see if they have been modified. Increasing
|
|
this interval will decrease the number of system calls we make, but
|
|
increase the time to detect new log lines.</p>
|
|
|
|
<h4>
|
|
<a name="tags">
|
|
tags
|
|
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> Value type is <a href="../configuration#array">array</a> </li>
|
|
<li> There is no default value for this setting. </li>
|
|
</ul>
|
|
|
|
<p>Add any number of arbitrary tags to your event.</p>
|
|
|
|
<p>This can help with processing later.</p>
|
|
|
|
<h4>
|
|
<a name="type">
|
|
type
|
|
|
|
</a>
|
|
</h4>
|
|
|
|
<ul>
|
|
<li> Value type is <a href="../configuration#string">string</a> </li>
|
|
<li> There is no default value for this setting. </li>
|
|
</ul>
|
|
|
|
<p>Add a 'type' field to all events handled by this input.</p>
|
|
|
|
<p>Types are used mainly for filter activation.</p>
|
|
|
|
<p>If you create an input with type "foobar", then only filters
|
|
which also have type "foobar" will act on them.</p>
|
|
|
|
<p>The type is also stored as part of the event itself, so you
|
|
can also use the type to search for in the web interface.</p>
|
|
|
|
<p>If you try to set a type on an event that already has one (for
|
|
example when you send an event from a shipper to an indexer) then
|
|
a new input will not override the existing type. A type set at
|
|
the shipper stays with that event for its life even
|
|
when sent to another LogStash server.</p>
|
|
|
|
|
|
<hr>
|
|
|
|
This is documentation from <a href="https://github.com/logstash/logstash/blob/v1.2.0.beta1/lib/logstash/inputs/file.rb">lib/logstash/inputs/file.rb</a>
|