- support a pid file
- set $0 on our indexer & parser children
- handle INT and TERM signals (clean up children)
- switch to waitpid/WNOHANG/sleep(5) loop instead of Process.waitall because
of a lame bug with ruby 1.8.5 and SIGTERM
op = client.sendmsg("logstash-directory", msg) do |response|
puts response
:finished
end
op.wait_until_finished
puts "Done!"
If Socket#sendmsg is given a block, it will return an Operation that is
useful shown above.
- make @close work
- main @id queue cannot be :exclusive (nobody else can write)
- properly make topic exchange auto_delete and exclusive
- refactor search.rb client, start towards a common SearchClient class
(and work with amqp)
- remove sample agent.redhat.rb; sample config is the same thing
- indexers now broadcast to each other and build up a directory
- indexers respond to a directory request and provide a list of indexers
- move some errors/usage messages to $stderr
- refactor search.rb to query all available indexers
- Ignore dotfiles when considering files of grok patterns
- Fix our monkey patch adding String#strip_upper_ascii. This should probably
just be in another module and take a string as argument.
- Fix bin/search.rb to point at the right queue (indexer serves searches now)
- Fix websearch to point at correct queue.
hand-code. This method is as fast as the hand-coded method and with less
writing.
- Make message registration easier with a simple 'register' function:
class Foo < Message
register
end
We dynamically generate the message 'type' by the class name.
- make it easy to try using ruby marshal instead of json for wire format, for comparing speed.
- Set Thread::abort_on_exception so when threads die due to simple syntax or
name errors they don't die silently.
- Add basic search client
for an ack for that message before sending another. Work around this:
- batch up messages to sendmsg() and flush when there are more than 10 in the
queue or there has been more than 1 second since flushing and the queue is
non-empty
This increases the indexing rate by a factor of 6.
- Add LogStash::Net::NoSocket
- Add special 'signal' socketpair that is always used in select() as a reader
so that we can, from sendmsg(), notify select() that we should end and loop
again
- MessageSocketMux#connect now returns true if the connection succeeded. An
exception is thrown otherwise.
- Only include writers who have populated output streams.
- If we had a client receiver (via #connect(..)) before, but it went away,
assume the connection was destroyed/reset by some other means and that this is
an error to be handled by the client.
- Don't immediately add a socket to the @writers list when it's created.
- If _sendmsg has a socket that is nil, raise NoSocket. If _sendmsg is called
with a nil socket, and @receiver (created through #connect(...)) is also nil,
this is an error and we throw NoSocket.
- Delete @receiver if it is a socket that is being removed.
- Delete socket from @writers once we flush it's data out the socket.
- Raise MessageClientConnectionReset on EOF/IOError or ECONNRESET
- indexer: Sync index every 60 seconds
- Add alpha version of Agent. It watches /var/log/messages.
Tested:
- Both the agent and indexer server are capable of recovering from disconnections.
- searcher test code (sandbox/searchclient.rb) works while Agent is feeding
the indexer
