- Add LogStash::Net::NoSocket
- Add special 'signal' socketpair that is always used in select() as a reader
so that we can, from sendmsg(), notify select() that we should end and loop
again
- MessageSocketMux#connect now returns true if the connection succeeded. An
exception is thrown otherwise.
- Only include writers who have populated output streams.
- If we had a client receiver (via #connect(..)) before, but it went away,
assume the connection was destroyed/reset by some other means and that this is
an error to be handled by the client.
- Don't immediately add a socket to the @writers list when it's created.
- If _sendmsg has a socket that is nil, raise NoSocket. If _sendmsg is called
with a nil socket, and @receiver (created through #connect(...)) is also nil,
this is an error and we throw NoSocket.
- Delete @receiver if it is a socket that is being removed.
- Delete socket from @writers once we flush it's data out the socket.
- Raise MessageClientConnectionReset on EOF/IOError or ECONNRESET
- indexer: Sync index every 60 seconds
- Add alpha version of Agent. It watches /var/log/messages.
Tested:
- Both the agent and indexer server are capable of recovering from disconnections.
- searcher test code (sandbox/searchclient.rb) works while Agent is feeding
the indexer
MessageSocketMux, but the subclass names detail the intent much more clearly.
Later, we may want to move MessageSocketMux#listen to MessageServer, and
#connect to MessageClient.
- Some style fixings:
* All private methods are prefixed with 'private' on the preceding line.
* End all methods with 'end # def <method name>'
* Make methods private that should be private
* Add documentation to many methods
- Add array support to MessageSocketMux#_sendmsg (and thus #sendmsg). This lets
a client internally queue things and send multiple message to the output
queue at the same time.
- Add 'success?' method to ResponseMessage and IndexEventResponse.
- Raise exception if we try to read a message that is too large and is likely
incorrect/corrupt.
- Refactor MessageReader#each to be much more concise
- Added MessageReader#get (and private methods #ready? and #next_length)
ACK'd using the test client and server.
- Comment out old debugging stuff
- IO.select() with nil timeout (block until there is data)
- Split MessageSocketMux#remove into remove_writer and remove_reader becuase
quite often we only want to close one.
- In MessageReader, since we are doing buffered IO, we need to defer any system
EOFError exceptions until our buffer is exhausted.
- Lock around critical sections (message handling, sendmsg, etc)
Having some weird error that we ar ehandling closes incorrectly.
Nondetemrinistic behavior observed by the client. (Close before all messages recieved, etc)
to a socket.
- MessageSocketMux#run now will return if there's no work left to do (like if
our socket is dead)
- Updated client play code to quit when we get all messaged ack'd
client and server networking.
A SocketMux can handle any message type. The only difference between a client
and a server is who initiates the TCP connection.
Clients call SocketMux#connect
Servers call SocketMux#listen
- Only strip upper-ascii in message values that are strings
- Shuffle namespacing around. Flat LogStash to heirarchical LogStash::Net::...
- Add stub Indexer server
- handle all date stuff in Log, not the sub-classes
- use "encoding" and "type" properly, rather than "type" and "name"
- switch to having a group of Groks, and doing first-pattern-match
- make example config.rb a little easier to read
because JSON.dump assumes UTF-8
- server: when we decode a message, if we have a handler for <msgname>Handler,
we should call that handler with the message as an argument.
- load all pattern files under patterns/
- grok captures don't include @LINE, so put it there ourselves
- properly filter grok captures per comments
- tear out grok cmdline stuff
- switch to seconds since epoch for @DATE
- add firewalls pattern with initial netscreen session close pattern
- Add MessageStream.clear for wiping messages in the stream.
- Make socket reading faster and more reliable (since read(N) may return elss
than N bytes)
Fun stats, dumping lines of apache logs into sandbox/srv.rb
from sandbox/client.rb:
1421000 finished @ 163/sec => 8680.2 secs
1422000 finished @ 163/sec => 8678.5 secs
1423000 finished @ 163/sec => 8679.1 secs
8500 lines per second? That's about 2MB/sec. Not bad?
to the correct message instance (an IndexEvent request becomes an
IndexEventRequest instance, etc)
- We use some metaprogramming tricks to bind Message JSON fields to
functions, see BindToHash and hashbind in net/message.rb
- Protocol versioning is poor right now, but it is present.
- The server code is not well-written, yet. I just wanted real client/server
encode/decode testing.
- Add some play code to sandbox/
