Commit graph

  • 257c2de5d0 - actually, since @close is set, it's safe to end EM right here Pete Fritchman 2009-10-19 05:53:17 +00:00
  • e909d241ec - less verbose - make @close work - main @id queue cannot be :exclusive (nobody else can write) - properly make topic exchange auto_delete and exclusive - refactor search.rb client, start towards a common SearchClient class (and work with amqp) Pete Fritchman 2009-10-19 05:52:18 +00:00
  • 56aac96795 s/stomp/amqp/ Pete Fritchman 2009-10-19 04:36:06 +00:00
  • 27204e825b - remove some debugging cruft - remove sample agent.redhat.rb; sample config is the same thing - indexers now broadcast to each other and build up a directory - indexers respond to a directory request and provide a list of indexers - move some errors/usage messages to $stderr - refactor search.rb to query all available indexers Pete Fritchman 2009-10-18 23:38:35 +00:00
  • 51a0e8ce8a - Add yaml config for agent - Ignore dotfiles when considering files of grok patterns - Fix our monkey patch adding String#strip_upper_ascii. This should probably just be in another module and take a string as argument. - Fix bin/search.rb to point at the right queue (indexer serves searches now) - Fix websearch to point at correct queue. Jordan Sissel 2009-10-16 09:57:13 +00:00
  • 9d6d58f1d3 - Moved cgrok's svn to named 'grok'. Jordan Sissel 2009-09-29 02:02:11 +00:00
  • 25c06132ce - Add logparsed. Runs N processes with a parser for each process. Jordan Sissel 2009-09-21 06:43:33 +00:00
  • 9b4e10e757 - Split indexer and parsers Jordan Sissel 2009-09-21 06:43:08 +00:00
  • a592c6fa1a - Don't use GrokMatch#captures anymore, use GrokMatch#each_capture, as it is more efficient. With this new method, we only iterate over the captures once. Jordan Sissel 2009-09-21 06:35:50 +00:00
  • 1582a29ce9 - Add instructions for running the web interface Jordan Sissel 2009-09-19 10:09:53 +00:00
  • 6ac738d73c - don't pp entry on index - add output of log_type when searching (for debugging) Jordan Sissel 2009-09-19 10:09:43 +00:00
  • 3b8a9416b6 - Add cron patterns to linux-syslog Jordan Sissel 2009-09-19 10:06:49 +00:00
  • 82e5bf79e9 - Don't hack an exception onto String#scan (only needed it for debugging anyway) Jordan Sissel 2009-09-19 10:06:31 +00:00
  • 40fbf7a670 - Correctly set defaults for limit and offset - Allow user to specify log_type - Fix next/prev urls - Escape form values when displaying them Jordan Sissel 2009-09-19 10:05:59 +00:00
  • dc4cc2088a - Make search results have next/prev links and show current position - Use GET instead of POST Jordan Sissel 2009-09-19 09:06:44 +00:00
  • 0c87bf740f - set / route to /search Jordan Sissel 2009-09-19 08:40:00 +00:00
  • cf8ee9b312 - Add logstash web frontend. This is a Merb application. Jordan Sissel 2009-09-19 08:37:04 +00:00
  • 0061adadc3 - config.yaml -> logstashd.yaml Jordan Sissel 2009-09-14 08:19:44 +00:00
  • 4b870cc9fd - add haproxy bits Jordan Sissel 2009-09-11 10:54:57 +00:00
  • e9c314dee6 - Move from config.rb to config.yaml - logstashd now takes a config.yaml as the only argument. Jordan Sissel 2009-09-11 10:24:13 +00:00
  • 9df61fc0c1 Updated installation/running instructions. Created an agent with sane defaults for redhat-based systems. Greg Retkowski 2009-09-10 21:24:20 +00:00
  • 13a54a1620 Fixed bug where server would crash if an invalid logtype was supplied to search.rb. Instead server now returns zero results. Greg Retkowski 2009-09-10 21:03:24 +00:00
  • fac8def339 - don't use stats anymore Jordan Sissel 2009-09-10 17:58:41 +00:00
  • cee6acd8e1 - Add missing messages Jordan Sissel 2009-09-10 17:58:12 +00:00
  • 034272e0f9 - remove debug line Jordan Sissel 2009-09-10 10:01:48 +00:00
  • 434a633013 - hashbind is back! Dynamically generate methods that we would normally hand-code. This method is as fast as the hand-coded method and with less writing. - Make message registration easier with a simple 'register' function: class Foo < Message register end Jordan Sissel 2009-09-10 09:59:24 +00:00
  • 07ade32cc9 - Add 'SearchHits' request to get the count of results (but not the results) for a given search query. - Allow a client to close in a message handler, but only handle the actual close until after we have ack'd the message. Jordan Sissel 2009-09-08 09:51:35 +00:00
  • d6cf0f34fb - ruby-prof showed that hashbind was taking the most time, so convert away from it. - make it easy to try using ruby marshal instead of json for wire format, for comparing speed. - Set Thread::abort_on_exception so when threads die due to simple syntax or name errors they don't die silently. - Add basic search client Jordan Sissel 2009-09-08 06:05:09 +00:00
  • f55513ba12 * In client ack mode, StompServer will only send one message at a time, waiting for an ack for that message before sending another. Work around this: - batch up messages to sendmsg() and flush when there are more than 10 in the queue or there has been more than 1 second since flushing and the queue is non-empty Jordan Sissel 2009-09-06 23:38:04 +00:00
  • 1e33e3af19 - Got things happy using STOMP Jordan Sissel 2009-09-06 22:13:06 +00:00
  • 42b767fc85 - move to use STOMP instead of our own silly messaging protocol. STOMP has: * message ids * reply support * message persistence until ack Jordan Sissel 2009-09-06 10:37:10 +00:00
  • 22b8c6eaca - don't flush stdout Jordan Sissel 2009-08-19 00:17:10 +00:00
  • a5c0f2e41d - Add more gems to INSTALL - Make epoll enableable via USE_EPOLL environ Jordan Sissel 2009-08-19 00:13:11 +00:00
  • 7d9a4dde56 - Remove more pre-eventmachine code Jordan Sissel 2009-08-18 19:48:21 +00:00
  • 46c4feb300 - Support ruby 1.9 Jordan Sissel 2009-08-18 19:47:05 +00:00
  • 7b83c35eaf - Don't use EventMachine.defer for running message handlers. Turns out it's faster if we don't push the call off to another thread. Jordan Sissel 2009-08-18 09:48:29 +00:00
  • 42df5ba36b - Support yielding to blocks for client and server #run methods - Start hacking on having client writes block if we are far behind on getting message acks (w/ exponential backoff). Jordan Sissel 2009-08-18 09:45:02 +00:00
  • 173840a827 - Refactor network code to use EventMachine Message decoding is working, but I think we're silently dropping data somewhere. Jordan Sissel 2009-08-18 09:01:39 +00:00
  • 35c8fe2670 - Add offset and limit paramters to SearchRequest Jordan Sissel 2009-08-17 09:37:35 +00:00
  • 788781bf22 - Call the message handler in a separate thread so we can stream responses (if there are multiple responses) back to the requestor as they are generated. This allows a large search to stream data back while the search occurs. - Do searching in chunks of 50 results at a time so we can send results back to to the requesting client in parallel with the remainder of the search. - Emit count of messages seen for debugging (will remove later) Jordan Sissel 2009-08-17 09:32:11 +00:00
  • 96e4881de2 - Fix Log#index_dir - SYNCDELAY for indexing to 10 seconds, for debugging/testing. - Batch search results to 10 per message block Jordan Sissel 2009-08-17 08:47:05 +00:00
  • 14d65e1522 - s/LOGSTASH_HOME/LOGSTASH_DIR/ - Fix variable references - Add 'finished' field to SearchResponse to indicate the end of search results. Jordan Sissel 2009-08-17 08:24:54 +00:00
  • 657266316d - Default statefile location of ~/.rb_since - Handle case where statefile does not exist Jordan Sissel 2009-08-17 08:23:48 +00:00
  • 1b1933c0f5 - Add checksumming to message payloads. This increases the message payload header size from 4 to 8. 4 bytes for a checksum; currently adler32. Checksum is only done on thes message content, not the header bytes. - Add String#adler32 (and as an alias, String#checksum) from Zlib.adler32 Jordan Sissel 2009-08-17 07:59:34 +00:00
  • bf69f3cc03 - make message_count a method Jordan Sissel 2009-08-17 07:10:09 +00:00
  • e43f7ccba3 - Add File::Tail::Since to get since(1)-like features on top of File::Tail Jordan Sissel 2009-08-17 06:33:30 +00:00
  • ad7c9b3f24 - remove old indexer prototype; the sandbox server is working now - move sandbox server to bin/logstashd.rb Pete Fritchman 2009-08-17 02:55:14 +00:00
  • e8efeea14a - move on-disk indexes to $LOGSTASH_HOME/var/indexes/log:type - move index creation into the ::Log module - calculate the next synctime after we run our .commits Pete Fritchman 2009-08-17 02:53:56 +00:00
  • f8c0627acd - Move MessageStream class to it's own file Jordan Sissel 2009-08-16 18:35:29 +00:00
  • 6af4e677a4 - Rather than having message handlers return a single message response, let's support streaming multiple responses for a request and yield messages instead of returning them. Jordan Sissel 2009-08-14 09:31:45 +00:00
  • 6cdb98559a - Add LogStash::Net::MessageClientConnectionReset - Add LogStash::Net::NoSocket - Add special 'signal' socketpair that is always used in select() as a reader so that we can, from sendmsg(), notify select() that we should end and loop again - MessageSocketMux#connect now returns true if the connection succeeded. An exception is thrown otherwise. - Only include writers who have populated output streams. - If we had a client receiver (via #connect(..)) before, but it went away, assume the connection was destroyed/reset by some other means and that this is an error to be handled by the client. - Don't immediately add a socket to the @writers list when it's created. - If _sendmsg has a socket that is nil, raise NoSocket. If _sendmsg is called with a nil socket, and @receiver (created through #connect(...)) is also nil, this is an error and we throw NoSocket. - Delete @receiver if it is a socket that is being removed. - Delete socket from @writers once we flush it's data out the socket. - Raise MessageClientConnectionReset on EOF/IOError or ECONNRESET - indexer: Sync index every 60 seconds - Add alpha version of Agent. It watches /var/log/messages. Jordan Sissel 2009-08-14 09:05:44 +00:00
  • ac4298f2d8 - clean up whitespace Jordan Sissel 2009-08-14 07:38:48 +00:00
  • f6df3a60ec - Add stub for success? method to ResponseMessage Jordan Sissel 2009-08-14 07:38:30 +00:00
  • 3b657e8e69 - Add MessageClient and MessageServer. These are just empty subclasses of MessageSocketMux, but the subclass names detail the intent much more clearly. Later, we may want to move MessageSocketMux#listen to MessageServer, and #connect to MessageClient. - Some style fixings: * All private methods are prefixed with 'private' on the preceding line. * End all methods with 'end # def <method name>' * Make methods private that should be private * Add documentation to many methods - Add array support to MessageSocketMux#_sendmsg (and thus #sendmsg). This lets a client internally queue things and send multiple message to the output queue at the same time. - Add 'success?' method to ResponseMessage and IndexEventResponse. - Raise exception if we try to read a message that is too large and is likely incorrect/corrupt. - Refactor MessageReader#each to be much more concise - Added MessageReader#get (and private methods #ready? and #next_length) Jordan Sissel 2009-08-14 07:19:36 +00:00
  • ff21452966 Slightly improved INSTALL a few fixes to module references require rubygems to get at json gem Greg Retkowski 2009-08-14 06:23:59 +00:00
  • 59279531b3 - Add Search{Message,Response} - Support SearchRequest in indexer server - Add playtest searchclient.rb Jordan Sissel 2009-08-12 06:17:58 +00:00
  • af54d862d6 - sandbox client should ping and try to index log lines - remove blank line in srv.rb Jordan Sissel 2009-08-11 07:11:22 +00:00
  • f5da7b3420 - Fully remove a sock if we get EPIPE or ECONNRESET - Add Ping{Request,Response} message - Better MessageReader buffer handling (handle multiple message blocks if we can) Jordan Sissel 2009-08-11 07:11:01 +00:00
  • 3ca59b3dbb - Make the Indexer server actually index logs - Also add ping handling (this should be refactored into another class that can be included to add ping support to any server) Jordan Sissel 2009-08-11 07:09:18 +00:00
  • e2d87ac44b - Reliable networking stuff again. I can send 10000 messages and get them all ACK'd using the test client and server. - Comment out old debugging stuff - IO.select() with nil timeout (block until there is data) - Split MessageSocketMux#remove into remove_writer and remove_reader becuase quite often we only want to close one. - In MessageReader, since we are doing buffered IO, we need to defer any system EOFError exceptions until our buffer is exhausted. Jordan Sissel 2009-08-10 09:00:16 +00:00
  • ed69d85b19 - Generate message ID only when needed - Lock around critical sections (message handling, sendmsg, etc) Jordan Sissel 2009-08-10 07:43:10 +00:00
  • ab6c2e0177 - Add MessageReader class for extrapolating network stream parsing - Move message ID sequencing to RequestMessage (Responses shouldn't generate new IDs under the current protocol) Jordan Sissel 2009-08-10 07:11:20 +00:00
  • 3bff2c835d - remove unused module Jordan Sissel 2009-08-10 04:35:23 +00:00
  • 216673ddd3 - update comment Jordan Sissel 2009-08-10 04:34:32 +00:00
  • 70fe2e37bf - comment out debugging Jordan Sissel 2009-08-10 03:41:32 +00:00
  • 005ae3a41c - Buffer messages with sendmsg() and flush them when select() says we can write to a socket. - MessageSocketMux#run now will return if there's no work left to do (like if our socket is dead) - Updated client play code to quit when we get all messaged ack'd Jordan Sissel 2009-08-10 03:40:57 +00:00
  • 716b15f002 - Comment-out some debug lines Jordan Sissel 2009-08-10 02:31:01 +00:00
  • 539e58d006 - Refactor ::Net::Server into ::Net::SocketMux. This class is used for both client and server networking. Jordan Sissel 2009-08-10 02:29:25 +00:00
  • 40f830d85a - capture pid in SYSLOGPROG - add some linux-syslog patterns - sample config for linux-syslog Pete Fritchman 2009-08-10 02:13:05 +00:00
  • 68183f9c6a - revive test_parse_entry - LOGSTASH_HOME - don't display @LINE Pete Fritchman 2009-08-10 02:12:13 +00:00
  • ee69c120c7 - OK, so I'm crazy. File load order does not matter, and it shouldn't. Pete Fritchman 2009-08-10 02:05:58 +00:00
  • 3d1745efc4 - work around a weird grok bug (?), load patterns in sort order, and always load the grok-patterns first Pete Fritchman 2009-08-10 01:59:51 +00:00
  • 064721b299 - refactor Log's required/optional keys for less duplication - handle all date stuff in Log, not the sub-classes - use "encoding" and "type" properly, rather than "type" and "name" - switch to having a group of Groks, and doing first-pattern-match - make example config.rb a little easier to read Pete Fritchman 2009-08-10 00:42:48 +00:00
  • 937bfb3c11 - When setting message values, always strip upper ascii (byte >= 128) values because JSON.dump assumes UTF-8 - server: when we decode a message, if we have a handler for <msgname>Handler, we should call that handler with the message as an argument. Jordan Sissel 2009-08-10 00:18:14 +00:00
  • d5c3a36087 - move everything to the LogStash namespace Pete Fritchman 2009-08-09 23:29:44 +00:00
  • f622532240 - move grok-patterns to patterns/ subdir - load all pattern files under patterns/ - grok captures don't include @LINE, so put it there ourselves - properly filter grok captures per comments - tear out grok cmdline stuff - switch to seconds since epoch for @DATE - add firewalls pattern with initial netscreen session close pattern Pete Fritchman 2009-08-09 20:40:36 +00:00
  • 71c6cb138f - all reflected in wiki/Design now Pete Fritchman 2009-08-09 18:43:00 +00:00
  • c0fd574685 - Read larger chunks into our buffer when reading messages - Make the client read from a file and dump 20 messages at a time to the server. Jordan Sissel 2009-08-09 11:07:48 +00:00
  • c79ed129cf - MessageStream should keep a count of its messages - Add MessageStream.clear for wiping messages in the stream. - Make socket reading faster and more reliable (since read(N) may return elss than N bytes) Jordan Sissel 2009-08-09 10:45:12 +00:00
  • 45663365a5 - Start work on network layer. Messages are automagically decoded off the wire to the correct message instance (an IndexEvent request becomes an IndexEventRequest instance, etc) - We use some metaprogramming tricks to bind Message JSON fields to functions, see BindToHash and hashbind in net/message.rb - Protocol versioning is poor right now, but it is present. - The server code is not well-written, yet. I just wanted real client/server encode/decode testing. - Add some play code to sandbox/ Jordan Sissel 2009-08-09 09:40:57 +00:00
  • e7a6f57274 - respect LOGSTASH_HOME Pete Fritchman 2009-08-09 03:30:17 +00:00
  • a6f282c4d7 - let ferret index Arrays, it does the right thing. Pete Fritchman 2009-08-09 02:28:51 +00:00
  • 3d9cf8e58f - add profiling (temporary, but very useful for now) Pete Fritchman 2009-08-07 00:52:17 +00:00
  • 127394a84b - Use RubyGrok instead of IO.popen("grok ...") Requires installed: http://semicomplete.googlecode.com/svn/cgrok/ruby/ - use $HOME for logstash index directory. - Have import use Time.now.to_f to get higher-precision time values. Jordan Sissel 2009-08-06 08:39:05 +00:00
  • b92be62794 - remove the grok teardown stuff, since we're going to be using cgrok's new ruby api - ignore fields not explicitly named in grok (i.e. %{FOO} is ignored, but %{FOO:bar} is imported as key=bar) Pete Fritchman 2009-08-06 05:50:52 +00:00
  • 002c080abb - display import rate Pete Fritchman 2009-08-06 05:38:20 +00:00
  • 944939a7ee - search by @DATE, by default Pete Fritchman 2009-08-06 05:37:59 +00:00
  • 94944089d9 - actually create an index from the default FieldInfos. now that we're using these defaults, importing is faster and there is a much better log:index size ratio. Pete Fritchman 2009-08-06 05:37:23 +00:00
  • 82fdc11f2e - early early early-stage logstash prototype Pete Fritchman 2009-08-05 01:01:23 +00:00