- make @close work
- main @id queue cannot be :exclusive (nobody else can write)
- properly make topic exchange auto_delete and exclusive
- refactor search.rb client, start towards a common SearchClient class
(and work with amqp)
- remove sample agent.redhat.rb; sample config is the same thing
- indexers now broadcast to each other and build up a directory
- indexers respond to a directory request and provide a list of indexers
- move some errors/usage messages to $stderr
- refactor search.rb to query all available indexers
- Ignore dotfiles when considering files of grok patterns
- Fix our monkey patch adding String#strip_upper_ascii. This should probably
just be in another module and take a string as argument.
- Fix bin/search.rb to point at the right queue (indexer serves searches now)
- Fix websearch to point at correct queue.
hand-code. This method is as fast as the hand-coded method and with less
writing.
- Make message registration easier with a simple 'register' function:
class Foo < Message
register
end
We dynamically generate the message 'type' by the class name.
- make it easy to try using ruby marshal instead of json for wire format, for comparing speed.
- Set Thread::abort_on_exception so when threads die due to simple syntax or
name errors they don't die silently.
- Add basic search client
for an ack for that message before sending another. Work around this:
- batch up messages to sendmsg() and flush when there are more than 10 in the
queue or there has been more than 1 second since flushing and the queue is
non-empty
This increases the indexing rate by a factor of 6.
there are multiple responses) back to the requestor as they are generated.
This allows a large search to stream data back while the search occurs.
- Do searching in chunks of 50 results at a time so we can send results back to
to the requesting client in parallel with the remainder of the search.
- Emit count of messages seen for debugging (will remove later)
This increases the message payload header size from 4 to 8. 4 bytes for a
checksum; currently adler32. Checksum is only done on thes message content,
not the header bytes.
- Add String#adler32 (and as an alias, String#checksum) from Zlib.adler32
- Add LogStash::Net::NoSocket
- Add special 'signal' socketpair that is always used in select() as a reader
so that we can, from sendmsg(), notify select() that we should end and loop
again
- MessageSocketMux#connect now returns true if the connection succeeded. An
exception is thrown otherwise.
- Only include writers who have populated output streams.
- If we had a client receiver (via #connect(..)) before, but it went away,
assume the connection was destroyed/reset by some other means and that this is
an error to be handled by the client.
- Don't immediately add a socket to the @writers list when it's created.
- If _sendmsg has a socket that is nil, raise NoSocket. If _sendmsg is called
with a nil socket, and @receiver (created through #connect(...)) is also nil,
this is an error and we throw NoSocket.
- Delete @receiver if it is a socket that is being removed.
- Delete socket from @writers once we flush it's data out the socket.
- Raise MessageClientConnectionReset on EOF/IOError or ECONNRESET
- indexer: Sync index every 60 seconds
- Add alpha version of Agent. It watches /var/log/messages.
Tested:
- Both the agent and indexer server are capable of recovering from disconnections.
- searcher test code (sandbox/searchclient.rb) works while Agent is feeding
the indexer
