github-mirrors/logstash
1
0
Fork 0
mirror of https://github.com/elastic/logstash.git synced 2025-04-22 05:37:21 -04:00
Code Issues Projects Releases Packages Wiki Activity
3200 commits 254 branches 379 tags 173 MiB
Commit graph

3200 commits

This branch
This branch
All branches
Author SHA1 Message Date
Pete Fritchman
d4459ebf46 - consolidate to one logstashd that forks parsers & indexers 
- use logger for output
- stop handling QuitRequests
- re-up broadcast interval to 30 (was @ 5 for debugging)
- add debug output (logstashd -d ...)
 2009-10-19 19:03:49 +00:00
Jordan Sissel
e8159ef438 - remove debugging 2009-10-19 10:21:59 +00:00
Jordan Sissel
bff4949129 - responses should always have the same id as the request. 2009-10-19 10:21:49 +00:00
Jordan Sissel
405a813f68 - Move search's Directory querying to be synchronous 2009-10-19 10:19:49 +00:00
Jordan Sissel
e1839d359d - Add ability to do synchronous-like messaging: 
op = client.sendmsg("logstash-directory", msg) do |response|
    puts response
    :finished
  end

  op.wait_until_finished
  puts "Done!"

  If Socket#sendmsg is given a block, it will return an Operation that is
  useful shown above.
 2009-10-19 10:06:39 +00:00
Jordan Sissel
cd4baf78a6 - move amqp/em into a separate thread. 
- Things wanting to block until amqp/em are finished will still call
  Socket#run, which now only blocks until the new amqp/em thread complete (with
  Thread#join)
 2009-10-19 08:57:43 +00:00
Pete Fritchman
4afa19646b - include log_type in a LogKeysResponse message 2009-10-19 07:42:10 +00:00
Pete Fritchman
4755c0ae1c - add ability to query what keys are available for a log type 
- cache indexreaders, searchers, and queryparsers
 2009-10-19 07:09:07 +00:00
Pete Fritchman
6c4c36cdd7 - support querying what log types are available 2009-10-19 06:56:07 +00:00
Pete Fritchman
344e1a4aef - send back [@DATE, @LINE] in search results so we can collate results 
from multiple indexers
 2009-10-19 06:27:31 +00:00
Pete Fritchman
eeeb7b4cd3 - AMQPize. 2009-10-19 06:23:09 +00:00
Pete Fritchman
257c2de5d0 - actually, since @close is set, it's safe to end EM right here 2009-10-19 05:53:17 +00:00
Pete Fritchman
e909d241ec - less verbose 
- make @close work
- main @id queue cannot be :exclusive (nobody else can write)
- properly make topic exchange auto_delete and exclusive
- refactor search.rb client, start towards a common SearchClient class
  (and work with amqp)
 2009-10-19 05:52:18 +00:00
Pete Fritchman
56aac96795 s/stomp/amqp/ 2009-10-19 04:36:06 +00:00
Pete Fritchman
27204e825b - remove some debugging cruft 
- remove sample agent.redhat.rb; sample config is the same thing
- indexers now broadcast to each other and build up a directory
- indexers respond to a directory request and provide a list of indexers
- move some errors/usage messages to $stderr
- refactor search.rb to query all available indexers
 2009-10-18 23:38:35 +00:00
Jordan Sissel
51a0e8ce8a - Add yaml config for agent 
- Ignore dotfiles when considering files of grok patterns
- Fix our monkey patch adding String#strip_upper_ascii. This should probably
  just be in another module and take a string as argument.
- Fix bin/search.rb to point at the right queue (indexer serves searches now)
- Fix websearch to point at correct queue.
 2009-10-16 09:57:13 +00:00
Jordan Sissel
9d6d58f1d3 - Moved cgrok's svn to named 'grok'. 2009-09-29 02:02:11 +00:00
Jordan Sissel
25c06132ce - Add logparsed. Runs N processes with a parser for each process. 2009-09-21 06:43:33 +00:00
Jordan Sissel
9b4e10e757 - Split indexer and parsers 2009-09-21 06:43:08 +00:00
Jordan Sissel
a592c6fa1a - Don't use GrokMatch#captures anymore, use GrokMatch#each_capture, as it is 
more efficient. With this new method, we only iterate over the captures once.
 2009-09-21 06:35:50 +00:00
Jordan Sissel
1582a29ce9 - Add instructions for running the web interface 2009-09-19 10:09:53 +00:00
Jordan Sissel
6ac738d73c - don't pp entry on index 
- add output of log_type when searching (for debugging)
 2009-09-19 10:09:43 +00:00
Jordan Sissel
3b8a9416b6 - Add cron patterns to linux-syslog 2009-09-19 10:06:49 +00:00
Jordan Sissel
82e5bf79e9 - Don't hack an exception onto String#scan (only needed it for debugging anyway) 2009-09-19 10:06:31 +00:00
Jordan Sissel
40fbf7a670 - Correctly set defaults for limit and offset 
- Allow user to specify log_type
- Fix next/prev urls
- Escape form values when displaying them
 2009-09-19 10:05:59 +00:00
Jordan Sissel
dc4cc2088a - Make search results have next/prev links and show current position 
- Use GET instead of POST
 2009-09-19 09:06:44 +00:00
Jordan Sissel
0c87bf740f - set / route to /search 2009-09-19 08:40:00 +00:00
Jordan Sissel
cf8ee9b312 - Add logstash web frontend. This is a Merb application. 2009-09-19 08:37:04 +00:00
Jordan Sissel
0061adadc3 - config.yaml -> logstashd.yaml 2009-09-14 08:19:44 +00:00
Jordan Sissel
4b870cc9fd - add haproxy bits 2009-09-11 10:54:57 +00:00
Jordan Sissel
e9c314dee6 - Move from config.rb to config.yaml 
- logstashd now takes a config.yaml as the only argument.
 2009-09-11 10:24:13 +00:00
Greg Retkowski
9df61fc0c1 Updated installation/running instructions. Created an agent with sane 
defaults for redhat-based systems.
 2009-09-10 21:24:20 +00:00
Greg Retkowski
13a54a1620 Fixed bug where server would crash if an invalid logtype was supplied to 
search.rb. Instead server now returns zero results.
 2009-09-10 21:03:24 +00:00
Jordan Sissel
fac8def339 - don't use stats anymore 2009-09-10 17:58:41 +00:00
Jordan Sissel
cee6acd8e1 - Add missing messages 2009-09-10 17:58:12 +00:00
Jordan Sissel
034272e0f9 - remove debug line 2009-09-10 10:01:48 +00:00
Jordan Sissel
434a633013 - hashbind is back! Dynamically generate methods that we would normally 
hand-code. This method is as fast as the hand-coded method and with less
  writing.
- Make message registration easier with a simple 'register' function:
  class Foo < Message
    register
  end

  We dynamically generate the message 'type' by the class name.
 2009-09-10 09:59:24 +00:00
Jordan Sissel
07ade32cc9 - Add 'SearchHits' request to get the count of results (but not the results) 
for a given search query.
- Allow a client to close in a message handler, but only handle the actual
  close until after we have ack'd the message.
 2009-09-08 09:51:35 +00:00
Jordan Sissel
d6cf0f34fb - ruby-prof showed that hashbind was taking the most time, so convert away from it. 
- make it easy to try using ruby  marshal instead of json for wire format, for comparing speed.
- Set Thread::abort_on_exception so when threads die due to simple syntax or
  name errors they don't die silently.
- Add basic search client
 2009-09-08 06:05:09 +00:00
Jordan Sissel
f55513ba12 * In client ack mode, StompServer will only send one message at a time, waiting 
for an ack for that message before sending another. Work around this:
  - batch up messages to sendmsg() and flush when there are more than 10 in the
    queue or there has been more than 1 second since flushing and the queue is
    non-empty

This increases the indexing rate by a factor of 6.
 2009-09-06 23:38:04 +00:00
Jordan Sissel
1e33e3af19 - Got things happy using STOMP 2009-09-06 22:13:06 +00:00
Jordan Sissel
42b767fc85 - move to use STOMP instead of our own silly messaging protocol. STOMP has: 
* message ids
  * reply support
  * message persistence until ack

Not fully working yet.
 2009-09-06 10:37:10 +00:00
Jordan Sissel
22b8c6eaca - don't flush stdout 2009-08-19 00:17:10 +00:00
Jordan Sissel
a5c0f2e41d - Add more gems to INSTALL 
- Make epoll enableable via USE_EPOLL environ
 2009-08-19 00:13:11 +00:00
Jordan Sissel
7d9a4dde56 - Remove more pre-eventmachine code 2009-08-18 19:48:21 +00:00
Jordan Sissel
46c4feb300 - Support ruby 1.9 2009-08-18 19:47:05 +00:00
Jordan Sissel
7b83c35eaf - Don't use EventMachine.defer for running message handlers. Turns out it's 
faster if we don't push the call off to another thread.
 2009-08-18 09:48:29 +00:00
Jordan Sissel
42df5ba36b - Support yielding to blocks for client and server #run methods 
- Start hacking on having client writes block if we are far behind on getting
  message acks (w/ exponential backoff).
 2009-08-18 09:45:02 +00:00
Jordan Sissel
173840a827 - Refactor network code to use EventMachine 
Message decoding is working, but I think we're silently dropping data somewhere.
 2009-08-18 09:01:39 +00:00
Jordan Sissel
35c8fe2670 - Add offset and limit paramters to SearchRequest 2009-08-17 09:37:35 +00:00