github-mirrors/logstash
1
0
Fork 0
mirror of https://github.com/elastic/logstash.git synced 2025-04-21 13:18:16 -04:00
Code Issues Projects Releases Packages Wiki Activity
1149 commits 254 branches 379 tags 173 MiB
Commit graph

1149 commits

This branch
This branch
All branches
Author SHA1 Message Date
Jordan Sissel
7f66dcc2ea - use event machine timers 2010-04-11 15:51:36 +00:00
Jordan Sissel
e020e8487e - use elasticsearch instead of an indexer 2010-03-04 06:33:59 +00:00
Jordan Sissel
d84f742554 - can't include a class, only a module. Fix later. 2010-02-19 07:50:53 +00:00
Jordan Sissel
056a1d6920 - Move to use ElasticSearch's REST api for indexing instead of our own custom mqrpc+ferret instance. 2010-02-19 07:47:10 +00:00
Jordan Sissel
44d8eb4143 - Grab the pidfile before we daemonize so we can report pid lock errors 
promptly.
 2010-02-19 06:58:47 +00:00
Jordan Sissel
f079e8e2be - fix class usage. InvalidArgument isn't valid, we want ArgumentError 2010-01-26 09:05:29 +00:00
Pete Fritchman
4b3460491a - s/info/debug/ for "Sending" messages 2009-12-16 23:09:00 +00:00
Pete Fritchman
5143d3b85b - bump to 0.3.6 2009-11-30 22:12:23 +00:00
Pete Fritchman
468ae95ca2 - create a new thread for watching logs so we can properly handle globs. 
We don't just expand the glob on start, we constantly re-scan for any
  new log files.
 2009-11-15 05:35:26 +00:00
Pete Fritchman
c5448cf479 - detect when an index is corrupt (index.flush throws an exception), 
log a fatal error, and kill the entire logstashd process.
 2009-11-08 07:20:38 +00:00
Pete Fritchman
8b7b1b4623 - use proper variable name for termination callback block 2009-11-08 06:31:37 +00:00
Jordan Sissel
3a02fb6abd - migrate up some code 2009-11-08 06:14:46 +00:00
Jordan Sissel
1c5c6b8c03 - don't require a file we don't have naymore 2009-11-08 06:10:15 +00:00
Pete Fritchman
2a4e947c34 - don't start up a File::Tail::Since until the file exists 
- if File::Tail::Since does exit, we want to know about it
 2009-11-08 05:18:48 +00:00
Pete Fritchman
fff7413057 - gracefully handle errors from log parsers 
+ they can throw a LogStash::Log::LogParseError now
- handle JSON parse exceptions
 2009-11-08 05:07:12 +00:00
Jordan Sissel
e838dfaee4 - Remove old SlidingWindowSet code that MQRPC replaced with SizedThreadSafeHash 2009-11-07 23:43:39 +00:00
Jordan Sissel
8c22b1f314 - Style 2009-11-07 22:37:38 +00:00
Jordan Sissel
445eada3e2 - use Util::collapse 2009-11-07 22:36:39 +00:00
Jordan Sissel
b22506fa17 - style fix 2009-11-07 22:34:30 +00:00
Jordan Sissel
50eb137e1e - Use mqrpc's new 'delayable' message header 2009-11-07 21:58:40 +00:00
Pete Fritchman
f4b0fafc00 - bump to 0.3.5 (mqrpc split-out) 
- include wiki docs in rpm
 2009-11-07 21:37:29 +00:00
Jordan Sissel
c234e01f38 - Messages sent from parser -> indexer that came from an IndexEventRequest now 
get pushed into a queue that is managed by a separate thread. This is
  necessary to prevent sendmsg() from blocking the main AMQP reader thread when
  sendmsg() might block due to sliding window closure.

  This queue length is unchecked, however, the correct fix is to unsubscribe
  from the input (the AMQP queue) and only resubscribe once our queue has
  cleared a bit.
 2009-11-06 10:16:44 +00:00
Jordan Sissel
5e51250c20 - before forwarding the indexeventrequest to the indexer, generate a new 
message id for it.
 2009-11-06 09:39:39 +00:00
Jordan Sissel
ef35581319 - disable agent index request buffering for now (need to debug parser race/halting-bug condition) 
- set response code in Indexer's IndexEventRequestHandler
 2009-11-06 09:32:01 +00:00
Jordan Sissel
c9fa9c87af - BaseConfig subclasses MQRPC::Config now 
- Update servers and clients to use MQRPC properly
 2009-11-06 08:57:20 +00:00
Jordan Sissel
790b081a31 - Move all messages to use MQRPC 2009-11-06 08:26:47 +00:00
Jordan Sissel
1f85aad3d5 - Purge old network code now implemented by MQRPC 2009-11-06 08:17:13 +00:00
Jordan Sissel
6d41874029 - Version bump. 2009-10-29 07:03:08 +00:00
Jordan Sissel
1331382bac - Graph the current query hits over the past 24 hours from current time. 2009-10-29 06:58:28 +00:00
Jordan Sissel
8af13cdef5 - set body margin to 5px 2009-10-29 06:57:26 +00:00
Jordan Sissel
ea91012694 - Add flot 0.6 2009-10-29 06:56:29 +00:00
Jordan Sissel
20ce5e4acd - Add graphpoints.json template 2009-10-28 08:21:34 +00:00
Jordan Sissel
728221d326 - Add #searchhits to the search client. Takes a log type and array of queries, 
returns a hash of query => hitcount
- Fix LogStash::Operation behavior. If 'wait_until_finish' was called after the
  operation finished, not before, we would deadlock. Now any wait_until_finish
  call will succeed and return immediately if the operation has already finished.
  It will still block normally if the operation has not finished.
- Comment-out the sliding window stuff
- Add 'graphpoints' action for the web. Querying this will return an array of
  [timestamp_in_ms, hits] for the query for some period of timestamps for your
  query. Makes happy use of the Operation class so we can send a pile of search
  requests in parallel and wait until they finish. 24 queries (one for every
  hour) takes less than a second.
 2009-10-28 08:21:14 +00:00
Jordan Sissel
8b826e5215 - Attempt to use sliding windows 2009-10-26 18:54:40 +00:00
Jordan Sissel
0e68c317cb - Add my attempt at a locking 'sliding window' class that would prevent 
new additions if the 'want' buffer was full.
 2009-10-26 06:20:56 +00:00
Jordan Sissel
03707d1e20 - set durable on queues 2009-10-26 06:16:55 +00:00
Jordan Sissel
5dcd3ff22f - we should send a reply to the client sending the indexrequest 2009-10-26 06:14:11 +00:00
Jordan Sissel
473ee8cd74 - Add ':' to URIPATH 2009-10-25 09:46:00 +00:00
Jordan Sissel
4e9eb6ae28 - write File::Tail::Since state per file to ~/.rb_since_d/uid.filepath 
This relieves contention for the single ~/.rb_since file

  Also try to write to /var/run/rb_since/uid.filepath first, if we can.
 2009-10-25 09:41:36 +00:00
Jordan Sissel
c055692ca0 - Add 'first' link to search results 2009-10-25 09:13:21 +00:00
Jordan Sissel
cf6986d8e5 - Add 'last' link to jump to end of search results 
- Increase default 'limit' from 20 to 100.
 2009-10-25 09:11:24 +00:00
Jordan Sissel
c3ab5803ea - Add changelog 
- version to 0.3.3
 2009-10-25 04:54:29 +00:00
Jordan Sissel
a5886b21ff - reformat for readability 
- fix bug where 'prev' wasn't including log_type
 2009-10-25 04:49:14 +00:00
Jordan Sissel
9af8615fe0 - Support message buffering again, but now support both buffering and 
non-buffering (ie; send message now). This is akin to TCP_NODELAY but is
  configurable on a per-message basis by calling 'message.want_buffer(true)'
 2009-10-25 04:29:06 +00:00
Jordan Sissel
f35a1ea88a - Follow symlink if pattern_dir is one 2009-10-25 04:27:27 +00:00
Jordan Sissel
78cfbff565 - uri paths can have /[@{}]/ in it 2009-10-25 04:26:10 +00:00
Jordan Sissel
226027c024 - Disable message buffering internal to logstash. Previously, sendmsg() would 
push into an array which would get flushed when full or once per second.
  This was causing any normally-fast search request to take a minimum of 1 second.

  The original need for buffering was due to features of STOMP (not used
  anymore) that would limit the number of messages we could handle at once. We
  would buffer multiple messages to a single STOMP queue message so that we
  could handle multiple messages. This may not be a problem anymore with AMQP.

  Set BUFFER_DELAY_MESSAGES to true in socket.rb if you want to re-enable this.

- Add 'timestamp' attribute for all messages. This attribute is set
  automatically when it is sent with MessageSocket#sendmsg.
- Add blocking on MessageSocket#initialize to wait for the AMQP thread. Uses
  the same mutex/conditionvariable method we use to block LogStash::Operation
  instances.
- Removed old 'USE_MARSHAL' support as I think we're pretty happy with JSON now.
 2009-10-25 03:24:02 +00:00
Jordan Sissel
b9a273885f - search.rb use new LogStash::StopWatch class 2009-10-25 03:19:11 +00:00
Jordan Sissel
646a0948ed - Add LogStash::StopWatch class for recording execution time of code 2009-10-25 03:14:42 +00:00
Jordan Sissel
18476af99b - Only ever create one search client, and do it on app startup. 2009-10-25 03:11:04 +00:00