Commit graph

  • 8b7b1b4623 - use proper variable name for termination callback block Pete Fritchman 2009-11-08 06:31:37 +00:00
  • 3a02fb6abd - migrate up some code Jordan Sissel 2009-11-08 06:14:46 +00:00
  • 1c5c6b8c03 - don't require a file we don't have naymore Jordan Sissel 2009-11-08 06:10:15 +00:00
  • 2a4e947c34 - don't start up a File::Tail::Since until the file exists - if File::Tail::Since does exit, we want to know about it Pete Fritchman 2009-11-08 05:18:48 +00:00
  • fff7413057 - gracefully handle errors from log parsers + they can throw a LogStash::Log::LogParseError now - handle JSON parse exceptions Pete Fritchman 2009-11-08 05:07:12 +00:00
  • e838dfaee4 - Remove old SlidingWindowSet code that MQRPC replaced with SizedThreadSafeHash Jordan Sissel 2009-11-07 23:43:39 +00:00
  • 8c22b1f314 - Style Jordan Sissel 2009-11-07 22:37:38 +00:00
  • 445eada3e2 - use Util::collapse Jordan Sissel 2009-11-07 22:36:39 +00:00
  • b22506fa17 - style fix Jordan Sissel 2009-11-07 22:34:30 +00:00
  • 50eb137e1e - Use mqrpc's new 'delayable' message header Jordan Sissel 2009-11-07 21:58:40 +00:00
  • f4b0fafc00 - bump to 0.3.5 (mqrpc split-out) - include wiki docs in rpm Pete Fritchman 2009-11-07 21:37:29 +00:00
  • c234e01f38 - Messages sent from parser -> indexer that came from an IndexEventRequest now get pushed into a queue that is managed by a separate thread. This is necessary to prevent sendmsg() from blocking the main AMQP reader thread when sendmsg() might block due to sliding window closure. Jordan Sissel 2009-11-06 10:16:44 +00:00
  • 5e51250c20 - before forwarding the indexeventrequest to the indexer, generate a new message id for it. Jordan Sissel 2009-11-06 09:39:39 +00:00
  • ef35581319 - disable agent index request buffering for now (need to debug parser race/halting-bug condition) - set response code in Indexer's IndexEventRequestHandler Jordan Sissel 2009-11-06 09:32:01 +00:00
  • c9fa9c87af - BaseConfig subclasses MQRPC::Config now - Update servers and clients to use MQRPC properly Jordan Sissel 2009-11-06 08:57:20 +00:00
  • 790b081a31 - Move all messages to use MQRPC Jordan Sissel 2009-11-06 08:26:47 +00:00
  • 1f85aad3d5 - Purge old network code now implemented by MQRPC Jordan Sissel 2009-11-06 08:17:13 +00:00
  • 6d41874029 - Version bump. Jordan Sissel 2009-10-29 07:03:08 +00:00
  • 1331382bac - Graph the current query hits over the past 24 hours from current time. Jordan Sissel 2009-10-29 06:58:28 +00:00
  • 8af13cdef5 - set body margin to 5px Jordan Sissel 2009-10-29 06:57:26 +00:00
  • ea91012694 - Add flot 0.6 Jordan Sissel 2009-10-29 06:56:29 +00:00
  • 20ce5e4acd - Add graphpoints.json template Jordan Sissel 2009-10-28 08:21:34 +00:00
  • 728221d326 - Add #searchhits to the search client. Takes a log type and array of queries, returns a hash of query => hitcount - Fix LogStash::Operation behavior. If 'wait_until_finish' was called after the operation finished, not before, we would deadlock. Now any wait_until_finish call will succeed and return immediately if the operation has already finished. It will still block normally if the operation has not finished. - Comment-out the sliding window stuff - Add 'graphpoints' action for the web. Querying this will return an array of [timestamp_in_ms, hits] for the query for some period of timestamps for your query. Makes happy use of the Operation class so we can send a pile of search requests in parallel and wait until they finish. 24 queries (one for every hour) takes less than a second. Jordan Sissel 2009-10-28 08:21:14 +00:00
  • 8b826e5215 - Attempt to use sliding windows Jordan Sissel 2009-10-26 18:54:40 +00:00
  • 0e68c317cb - Add my attempt at a locking 'sliding window' class that would prevent new additions if the 'want' buffer was full. Jordan Sissel 2009-10-26 06:20:56 +00:00
  • 03707d1e20 - set durable on queues Jordan Sissel 2009-10-26 06:16:55 +00:00
  • 5dcd3ff22f - we should send a reply to the client sending the indexrequest Jordan Sissel 2009-10-26 06:14:11 +00:00
  • 473ee8cd74 - Add ':' to URIPATH Jordan Sissel 2009-10-25 09:46:00 +00:00
  • 4e9eb6ae28 - write File::Tail::Since state per file to ~/.rb_since_d/uid.filepath This relieves contention for the single ~/.rb_since file Jordan Sissel 2009-10-25 09:41:36 +00:00
  • c055692ca0 - Add 'first' link to search results Jordan Sissel 2009-10-25 09:13:21 +00:00
  • cf6986d8e5 - Add 'last' link to jump to end of search results - Increase default 'limit' from 20 to 100. Jordan Sissel 2009-10-25 09:11:24 +00:00
  • c3ab5803ea - Add changelog - version to 0.3.3 Jordan Sissel 2009-10-25 04:54:29 +00:00
  • a5886b21ff - reformat for readability - fix bug where 'prev' wasn't including log_type Jordan Sissel 2009-10-25 04:49:14 +00:00
  • 9af8615fe0 - Support message buffering again, but now support both buffering and non-buffering (ie; send message now). This is akin to TCP_NODELAY but is configurable on a per-message basis by calling 'message.want_buffer(true)' Jordan Sissel 2009-10-25 04:29:06 +00:00
  • f35a1ea88a - Follow symlink if pattern_dir is one Jordan Sissel 2009-10-25 04:27:27 +00:00
  • 78cfbff565 - uri paths can have /[@{}]/ in it Jordan Sissel 2009-10-25 04:26:10 +00:00
  • 226027c024 - Disable message buffering internal to logstash. Previously, sendmsg() would push into an array which would get flushed when full or once per second. This was causing any normally-fast search request to take a minimum of 1 second. Jordan Sissel 2009-10-25 03:24:02 +00:00
  • b9a273885f - search.rb use new LogStash::StopWatch class Jordan Sissel 2009-10-25 03:19:11 +00:00
  • 646a0948ed - Add LogStash::StopWatch class for recording execution time of code Jordan Sissel 2009-10-25 03:14:42 +00:00
  • 18476af99b - Only ever create one search client, and do it on app startup. Jordan Sissel 2009-10-25 03:11:04 +00:00
  • b5ff853c4a - remove file that doesn't need to be in svn Jordan Sissel 2009-10-25 03:10:42 +00:00
  • 4abe95037a - syslog program names can have periods or underscores Jordan Sissel 2009-10-23 16:57:27 +00:00
  • 9195abd6de - make get_ferret open a new IndexReader and Searcher every time, otherwise we can get stale search results. Jordan Sissel 2009-10-23 10:07:10 +00:00
  • 645d17da70 - version bump Jordan Sissel 2009-10-23 08:44:21 +00:00
  • 4de3175b57 - remove old sanbox we don't need anymore Jordan Sissel 2009-10-23 08:43:45 +00:00
  • 629c264113 - fix web search config yaml path - fix web dependencies Jordan Sissel 2009-10-23 08:42:17 +00:00
  • 807461e79e - set default / to /search Jordan Sissel 2009-10-23 08:12:30 +00:00
  • ddbf0ebe36 - Add new webapp which is of 'merb-gen core' lineage. Much fewer dependencies than previous version. Jordan Sissel 2009-10-23 08:11:44 +00:00
  • 5ce84cf45b - Remove 'web' as it requires too many things we don't use. Jordan Sissel 2009-10-23 08:10:53 +00:00
  • ddc06fd32b - Update INSTALL docs Jordan Sissel 2009-10-23 06:56:45 +00:00
  • b4b831ce9d - version bump for stomp removal Jordan Sissel 2009-10-21 08:32:13 +00:00
  • bb22cf63b7 - purge stomp Jordan Sissel 2009-10-21 08:30:45 +00:00
  • 168ca8cbf5 - include lib/clients/ in logstash-libs RPM Pete Fritchman 2009-10-21 07:10:27 +00:00
  • 4a3d5f2e8b - explicitly catch Ferret::FileNotFound Jordan Sissel 2009-10-21 02:47:53 +00:00
  • 04d2cee11e - fix pid/dying? logic - handle exceptions from get_ferret (trying to search unindexed log types) - die if we can't create a new index Pete Fritchman 2009-10-21 02:38:26 +00:00
  • af918884c9 - default search query includes 'AND -@NEEDSPARSING:1' unless we have @NEEDSPARSING in the query Jordan Sissel 2009-10-21 02:19:36 +00:00
  • 1b7e57b8d0 - Update Clients::Search to block on the search ops. Clients::Search#search now returns [hits, results_array] Jordan Sissel 2009-10-21 02:06:38 +00:00
  • a35abbbdad - factor out search client - make web and bin/search.rb use the new search client class Jordan Sissel 2009-10-21 01:58:42 +00:00
  • 15b830abc4 - use new BaseConfig for search tool - limit search results to 100 for now Jordan Sissel 2009-10-21 01:32:55 +00:00
  • f42fe931e1 - Use File.basename($0) Jordan Sissel 2009-10-21 01:03:47 +00:00
  • 875a6a9fee - make the agent use LogStash::Program Jordan Sissel 2009-10-21 01:02:38 +00:00
  • 21f04f4e8b - remove debugging - abort if @daemonize and no @logfile set Jordan Sissel 2009-10-21 01:00:26 +00:00
  • 4879992ab2 - add lib/program.rb to logstash-libs Pete Fritchman 2009-10-21 00:52:49 +00:00
  • a2c4c15918 - Add workaround for signal/syscall interruption bug in ruby 1.8.5 Jordan Sissel 2009-10-21 00:47:56 +00:00
  • 1ac4ed62a6 - facter out common program activities to LogStash::Program Jordan Sissel 2009-10-21 00:41:32 +00:00
  • 8b32b04aea - start using config values for connecting to the broker - print some warnings when we are reconnecting to the broker (could be indicative of bad MQ settings) Pete Fritchman 2009-10-21 00:36:11 +00:00
  • 3db7ad4a77 - handle pidfile better on shutdown - INT just calls TERM Pete Fritchman 2009-10-21 00:13:11 +00:00
  • dc27987d9e - ENV["LOGSTASH_DIR"] is deprecated. Pete Fritchman 2009-10-20 23:52:40 +00:00
  • 6d685bfdcf - RPM spec file Pete Fritchman 2009-10-20 21:57:00 +00:00
  • e5c9d75efd - fix init script Pete Fritchman 2009-10-20 21:56:19 +00:00
  • 9ad4e04f89 - open with happier mode (create if not exist, open readwrite) Jordan Sissel 2009-10-20 21:53:29 +00:00
  • f3ba581fcb - we want "w+" for the pidfile open because it may not exist yet Pete Fritchman 2009-10-20 21:24:41 +00:00
  • 282d153d11 - logstash-agent init script & configuration Pete Fritchman 2009-10-20 21:07:38 +00:00
  • 811ee9410c - make status & condrestart work Pete Fritchman 2009-10-20 21:07:28 +00:00
  • a6bb2b9405 - rename agent.rb to logstash-agent Pete Fritchman 2009-10-20 20:59:47 +00:00
  • 2edda3d34c - Add locking to our pidfile. Jordan Sissel 2009-10-20 20:59:08 +00:00
  • b9f811eea5 - fix pattern_dir support (again) - some defaults Pete Fritchman 2009-10-20 20:50:02 +00:00
  • 7e7dc90cfa - proper support for pattern_dir Pete Fritchman 2009-10-20 20:37:57 +00:00
  • a9bc1324f0 - If one of our children dies, send ourselves a SIGTERM and do a clean shutdown. We want our OS/watcher/whatever to know logstash is dead, rather than continue to run with random children dead. - Remove pidfile on shutdown Pete Fritchman 2009-10-20 20:35:16 +00:00
  • 4a70384313 - logfile option Pete Fritchman 2009-10-20 20:19:46 +00:00
  • 2711d2a6f6 - first pass at rhel init script & sample sysconfig Pete Fritchman 2009-10-20 20:13:53 +00:00
  • 24169ef165 - refactor agent to be a real daemon - refactor configs (all inherit from BaseConfig) - move beef of agent code to lib/net/clients/ - use optionparser's ".order" instead of ".parse" Pete Fritchman 2009-10-20 17:40:44 +00:00
  • 26082d9309 - Proper daemonizing - Add --logfile for directing STDOUT and STDERR to a file. - Require logfile if we are to daemonize - Set parent process $0 to 'logstash (supervisor)' Jordan Sissel 2009-10-20 06:59:02 +00:00
  • c151016413 - support daemonizing - support a pid file - set $0 on our indexer & parser children - handle INT and TERM signals (clean up children) - switch to waitpid/WNOHANG/sleep(5) loop instead of Process.waitall because of a lame bug with ruby 1.8.5 and SIGTERM Pete Fritchman 2009-10-20 04:55:34 +00:00
  • 8a332a01e2 - logstash_dir is where we write stuff. not necessarily where our patterns are installed, so allow pattern_dir. Pete Fritchman 2009-10-20 04:06:44 +00:00
  • ebbcb3697f - default location is /var/logstash Pete Fritchman 2009-10-20 03:53:06 +00:00
  • 33222b3ed4 - helper for building release tarball Pete Fritchman 2009-10-20 03:46:19 +00:00
  • f47a0aaf48 - move configs into etc/ Pete Fritchman 2009-10-20 03:45:19 +00:00
  • c02eb62bcc - remove hacks/testing files that we won't be packaging up Pete Fritchman 2009-10-20 01:20:58 +00:00
  • d4459ebf46 - consolidate to one logstashd that forks parsers & indexers - use logger for output - stop handling QuitRequests - re-up broadcast interval to 30 (was @ 5 for debugging) - add debug output (logstashd -d ...) Pete Fritchman 2009-10-19 19:03:49 +00:00
  • e8159ef438 - remove debugging Jordan Sissel 2009-10-19 10:21:59 +00:00
  • bff4949129 - responses should always have the same id as the request. Jordan Sissel 2009-10-19 10:21:49 +00:00
  • 405a813f68 - Move search's Directory querying to be synchronous Jordan Sissel 2009-10-19 10:19:49 +00:00
  • e1839d359d - Add ability to do synchronous-like messaging: Jordan Sissel 2009-10-19 10:06:39 +00:00
  • cd4baf78a6 - move amqp/em into a separate thread. - Things wanting to block until amqp/em are finished will still call Socket#run, which now only blocks until the new amqp/em thread complete (with Thread#join) Jordan Sissel 2009-10-19 08:57:43 +00:00
  • 4afa19646b - include log_type in a LogKeysResponse message Pete Fritchman 2009-10-19 07:42:10 +00:00
  • 4755c0ae1c - add ability to query what keys are available for a log type - cache indexreaders, searchers, and queryparsers Pete Fritchman 2009-10-19 07:09:07 +00:00
  • 6c4c36cdd7 - support querying what log types are available Pete Fritchman 2009-10-19 06:56:07 +00:00
  • 344e1a4aef - send back [@DATE, @LINE] in search results so we can collate results from multiple indexers Pete Fritchman 2009-10-19 06:27:31 +00:00
  • eeeb7b4cd3 - AMQPize. Pete Fritchman 2009-10-19 06:23:09 +00:00