Commit graph

10814 commits

Author SHA1 Message Date
Pete Fritchman
f622532240 - move grok-patterns to patterns/ subdir
- load all pattern files under patterns/
- grok captures don't include @LINE, so put it there ourselves
- properly filter grok captures per comments
- tear out grok cmdline stuff
- switch to seconds since epoch for @DATE
- add firewalls pattern with initial netscreen session close pattern
2009-08-09 20:40:36 +00:00
Pete Fritchman
71c6cb138f - all reflected in wiki/Design now 2009-08-09 18:43:00 +00:00
Jordan Sissel
c0fd574685 - Read larger chunks into our buffer when reading messages
- Make the client read from a file and dump 20 messages at a time to the server.
2009-08-09 11:07:48 +00:00
Jordan Sissel
c79ed129cf - MessageStream should keep a count of its messages
- Add MessageStream.clear for wiping messages in the stream.
- Make socket reading faster and more reliable (since read(N) may return elss
  than N bytes)

  Fun stats, dumping lines of apache logs into sandbox/srv.rb 
  from sandbox/client.rb:
    1421000 finished @ 163/sec => 8680.2 secs
    1422000 finished @ 163/sec => 8678.5 secs
    1423000 finished @ 163/sec => 8679.1 secs

  8500 lines per second? That's about 2MB/sec. Not bad?
2009-08-09 10:45:12 +00:00
Jordan Sissel
45663365a5 - Start work on network layer. Messages are automagically decoded off the wire
to the correct message instance (an IndexEvent request becomes an
  IndexEventRequest instance, etc)
- We use some metaprogramming tricks to bind Message JSON fields to
  functions, see BindToHash and hashbind in net/message.rb
- Protocol versioning is poor right now, but it is present.
- The server code is not well-written, yet. I just wanted real client/server
  encode/decode testing.
- Add some play code to sandbox/
2009-08-09 09:40:57 +00:00
Pete Fritchman
e7a6f57274 - respect LOGSTASH_HOME 2009-08-09 03:30:17 +00:00
Pete Fritchman
a6f282c4d7 - let ferret index Arrays, it does the right thing. 2009-08-09 02:28:51 +00:00
Pete Fritchman
3d9cf8e58f - add profiling (temporary, but very useful for now) 2009-08-07 00:52:17 +00:00
Jordan Sissel
127394a84b - Use RubyGrok instead of IO.popen("grok ...")
Requires installed: http://semicomplete.googlecode.com/svn/cgrok/ruby/
- use $HOME for logstash index directory.
- Have import use Time.now.to_f to get higher-precision time values.
2009-08-06 08:39:05 +00:00
Pete Fritchman
b92be62794 - remove the grok teardown stuff, since we're going to be using cgrok's
new ruby api
- ignore fields not explicitly named in grok (i.e. %{FOO} is ignored,
  but %{FOO:bar} is imported as key=bar)
2009-08-06 05:50:52 +00:00
Pete Fritchman
002c080abb - display import rate 2009-08-06 05:38:20 +00:00
Pete Fritchman
944939a7ee - search by @DATE, by default 2009-08-06 05:37:59 +00:00
Pete Fritchman
94944089d9 - actually create an index from the default FieldInfos. now that we're
using these defaults, importing is faster and there is a much better
  log:index size ratio.
2009-08-06 05:37:23 +00:00
Pete Fritchman
82fdc11f2e - early early early-stage logstash prototype 2009-08-05 01:01:23 +00:00