hand-code. This method is as fast as the hand-coded method and with less
writing.
- Make message registration easier with a simple 'register' function:
class Foo < Message
register
end
We dynamically generate the message 'type' by the class name.
- make it easy to try using ruby marshal instead of json for wire format, for comparing speed.
- Set Thread::abort_on_exception so when threads die due to simple syntax or
name errors they don't die silently.
- Add basic search client
for an ack for that message before sending another. Work around this:
- batch up messages to sendmsg() and flush when there are more than 10 in the
queue or there has been more than 1 second since flushing and the queue is
non-empty
This increases the indexing rate by a factor of 6.
there are multiple responses) back to the requestor as they are generated.
This allows a large search to stream data back while the search occurs.
- Do searching in chunks of 50 results at a time so we can send results back to
to the requesting client in parallel with the remainder of the search.
- Emit count of messages seen for debugging (will remove later)
This increases the message payload header size from 4 to 8. 4 bytes for a
checksum; currently adler32. Checksum is only done on thes message content,
not the header bytes.
- Add String#adler32 (and as an alias, String#checksum) from Zlib.adler32
- Add LogStash::Net::NoSocket
- Add special 'signal' socketpair that is always used in select() as a reader
so that we can, from sendmsg(), notify select() that we should end and loop
again
- MessageSocketMux#connect now returns true if the connection succeeded. An
exception is thrown otherwise.
- Only include writers who have populated output streams.
- If we had a client receiver (via #connect(..)) before, but it went away,
assume the connection was destroyed/reset by some other means and that this is
an error to be handled by the client.
- Don't immediately add a socket to the @writers list when it's created.
- If _sendmsg has a socket that is nil, raise NoSocket. If _sendmsg is called
with a nil socket, and @receiver (created through #connect(...)) is also nil,
this is an error and we throw NoSocket.
- Delete @receiver if it is a socket that is being removed.
- Delete socket from @writers once we flush it's data out the socket.
- Raise MessageClientConnectionReset on EOF/IOError or ECONNRESET
- indexer: Sync index every 60 seconds
- Add alpha version of Agent. It watches /var/log/messages.
Tested:
- Both the agent and indexer server are capable of recovering from disconnections.
- searcher test code (sandbox/searchclient.rb) works while Agent is feeding
the indexer
MessageSocketMux, but the subclass names detail the intent much more clearly.
Later, we may want to move MessageSocketMux#listen to MessageServer, and
#connect to MessageClient.
- Some style fixings:
* All private methods are prefixed with 'private' on the preceding line.
* End all methods with 'end # def <method name>'
* Make methods private that should be private
* Add documentation to many methods
- Add array support to MessageSocketMux#_sendmsg (and thus #sendmsg). This lets
a client internally queue things and send multiple message to the output
queue at the same time.
- Add 'success?' method to ResponseMessage and IndexEventResponse.
- Raise exception if we try to read a message that is too large and is likely
incorrect/corrupt.
- Refactor MessageReader#each to be much more concise
- Added MessageReader#get (and private methods #ready? and #next_length)
ACK'd using the test client and server.
- Comment out old debugging stuff
- IO.select() with nil timeout (block until there is data)
- Split MessageSocketMux#remove into remove_writer and remove_reader becuase
quite often we only want to close one.
- In MessageReader, since we are doing buffered IO, we need to defer any system
EOFError exceptions until our buffer is exhausted.
- Lock around critical sections (message handling, sendmsg, etc)
Having some weird error that we ar ehandling closes incorrectly.
Nondetemrinistic behavior observed by the client. (Close before all messages recieved, etc)
to a socket.
- MessageSocketMux#run now will return if there's no work left to do (like if
our socket is dead)
- Updated client play code to quit when we get all messaged ack'd
client and server networking.
A SocketMux can handle any message type. The only difference between a client
and a server is who initiates the TCP connection.
Clients call SocketMux#connect
Servers call SocketMux#listen
- Only strip upper-ascii in message values that are strings
- Shuffle namespacing around. Flat LogStash to heirarchical LogStash::Net::...
- Add stub Indexer server
