hand-code. This method is as fast as the hand-coded method and with less
writing.
- Make message registration easier with a simple 'register' function:
class Foo < Message
register
end
We dynamically generate the message 'type' by the class name.
- make it easy to try using ruby marshal instead of json for wire format, for comparing speed.
- Set Thread::abort_on_exception so when threads die due to simple syntax or
name errors they don't die silently.
- Add basic search client
for an ack for that message before sending another. Work around this:
- batch up messages to sendmsg() and flush when there are more than 10 in the
queue or there has been more than 1 second since flushing and the queue is
non-empty
This increases the indexing rate by a factor of 6.
there are multiple responses) back to the requestor as they are generated.
This allows a large search to stream data back while the search occurs.
- Do searching in chunks of 50 results at a time so we can send results back to
to the requesting client in parallel with the remainder of the search.
- Emit count of messages seen for debugging (will remove later)
This increases the message payload header size from 4 to 8. 4 bytes for a
checksum; currently adler32. Checksum is only done on thes message content,
not the header bytes.
- Add String#adler32 (and as an alias, String#checksum) from Zlib.adler32
- Add LogStash::Net::NoSocket
- Add special 'signal' socketpair that is always used in select() as a reader
so that we can, from sendmsg(), notify select() that we should end and loop
again
- MessageSocketMux#connect now returns true if the connection succeeded. An
exception is thrown otherwise.
- Only include writers who have populated output streams.
- If we had a client receiver (via #connect(..)) before, but it went away,
assume the connection was destroyed/reset by some other means and that this is
an error to be handled by the client.
- Don't immediately add a socket to the @writers list when it's created.
- If _sendmsg has a socket that is nil, raise NoSocket. If _sendmsg is called
with a nil socket, and @receiver (created through #connect(...)) is also nil,
this is an error and we throw NoSocket.
- Delete @receiver if it is a socket that is being removed.
- Delete socket from @writers once we flush it's data out the socket.
- Raise MessageClientConnectionReset on EOF/IOError or ECONNRESET
- indexer: Sync index every 60 seconds
- Add alpha version of Agent. It watches /var/log/messages.
Tested:
- Both the agent and indexer server are capable of recovering from disconnections.
- searcher test code (sandbox/searchclient.rb) works while Agent is feeding
the indexer
MessageSocketMux, but the subclass names detail the intent much more clearly.
Later, we may want to move MessageSocketMux#listen to MessageServer, and
#connect to MessageClient.
- Some style fixings:
* All private methods are prefixed with 'private' on the preceding line.
* End all methods with 'end # def <method name>'
* Make methods private that should be private
* Add documentation to many methods
- Add array support to MessageSocketMux#_sendmsg (and thus #sendmsg). This lets
a client internally queue things and send multiple message to the output
queue at the same time.
- Add 'success?' method to ResponseMessage and IndexEventResponse.
- Raise exception if we try to read a message that is too large and is likely
incorrect/corrupt.
- Refactor MessageReader#each to be much more concise
- Added MessageReader#get (and private methods #ready? and #next_length)
ACK'd using the test client and server.
- Comment out old debugging stuff
- IO.select() with nil timeout (block until there is data)
- Split MessageSocketMux#remove into remove_writer and remove_reader becuase
quite often we only want to close one.
- In MessageReader, since we are doing buffered IO, we need to defer any system
EOFError exceptions until our buffer is exhausted.
- Lock around critical sections (message handling, sendmsg, etc)
Having some weird error that we ar ehandling closes incorrectly.
Nondetemrinistic behavior observed by the client. (Close before all messages recieved, etc)
