Automatically choose most correct codec for bytestream inputs.

Now, if you try to use `plain` or `json` codecs with either `stdin` or
`tcp` inputs, it will automatically select `line` or `json_lines` codec
respectively.

Right now, many users try to use json codec with tcp input. This
*should* work, but due to technical limitations[1], it does not.
To compensate, we have always provided a json_lines codec to help
parse "streams of line-delimited json" which is what most folks
seem to be handling these days.

[1] I haven't found a JSON library capable of processing a byte stream.
Jackson has one, but it requires an input stream model which logstash
does not currently expose. Future work will make this possible.

Related: #1391
This commit is contained in:
Jordan Sissel 2014-05-20 04:52:58 +00:00
parent 59c4293ff1
commit 6480c9c89b
3 changed files with 18 additions and 0 deletions

View file

@ -118,4 +118,20 @@ class LogStash::Inputs::Base < LogStash::Plugin
event[field] = value
end
end
protected
def fix_streaming_codecs
require "logstash/codecs/plain"
require "logstash/codecs/line"
require "logstash/codecs/json"
require "logstash/codecs/json_lines"
case @codec
when LogStash::Codecs::Plain
@logger.info("Automatically switching from #{@codec.class.config_name} to line codec", :plugin => self.class.config_name)
@codec = LogStash::Codecs::Line.new
when LogStash::Codecs::JSON
@logger.info("Automatically switching from #{@codec.class.config_name} to json_lines codec", :plugin => self.class.config_name)
@codec = LogStash::Codecs::JSONLines.new
end
end
end # class LogStash::Inputs::Base

View file

@ -16,6 +16,7 @@ class LogStash::Inputs::Stdin < LogStash::Inputs::Base
public
def register
@host = Socket.gethostname
fix_streaming_codecs
end # def register
def run(queue)

View file

@ -59,6 +59,7 @@ class LogStash::Inputs::Tcp < LogStash::Inputs::Base
public
def register
fix_streaming_codecs
require "socket"
require "timeout"
require "openssl"