Commit graph

  • a579aea186 - Allow easy specification of ISO8601 format Jordan Sissel 2010-11-09 09:49:55 +00:00
  • 2953457843 - Add all patterns found Jordan Sissel 2010-11-09 09:27:57 +00:00
  • ec2ae60d83 add support for streaming logs to elasticsearch via a "rabbitmq river". we have to PUT a json config to the elasticsearch HTTP API to tell it where to look, then we use our amqp output to send bulk-data style index messages to elasticsearch. The ES side does bulk-queueing for us. Pete Fritchman 2010-11-01 01:46:20 +00:00
  • b250a0218c move CGI include around to proper place Pete Fritchman 2010-11-01 01:40:29 +00:00
  • 84ef233bca - parse query string on output URLs for additional options - amqp now properly supports "queue" type (and in general, works now) - amqp supports durable queues (?durable=1) - added amqp#receive_raw for publishing a string directly Pete Fritchman 2010-11-01 01:35:52 +00:00
  • 3014715d78 automatically include the right lib dir Pete Fritchman 2010-11-01 01:31:15 +00:00
  • 57f480adeb - Make TZ in DATESTAMP_OTHER optional - Add more debugging to filters/grok to make it easier to see pattern compile problems. - Add percentage progress to rate.sh (for tracking long imports on logstash) - Add content-type to the webserver - Update to web interface to use new @-prefixed notation (@message, etc) Jordan Sissel 2010-10-31 08:59:28 +00:00
  • b3e67888a1 Jordan Sissel 2010-10-31 08:52:41 +00:00
  • 8ef912a624 - Only push events to websockets when we have subscribers Jordan Sissel 2010-10-31 08:28:55 +00:00
  • 97d71ed130 - Add script to hackily check for progress on a long import. Jordan Sissel 2010-10-31 08:17:47 +00:00
  • 8462bd05fa introduce log types, in addition to tags Pete Fritchman 2010-10-31 06:01:46 +00:00
  • 7e80e89c39 show usage when we encouner option-related problems Pete Fritchman 2010-10-31 04:54:25 +00:00
  • 8d6419e191 s/throw/raise/ Pete Fritchman 2010-10-31 04:50:19 +00:00
  • f7ad0a0a9a create base classes for inputs/outputs/filters Pete Fritchman 2010-10-31 04:46:12 +00:00
  • 2aba74e2f4 - metadata uses '@' prefix notation. (@timestamp, @source, etc) Jordan Sissel 2010-10-31 02:22:12 +00:00
  • 0de9b9d9c3 - redirect / => /search - clean up sass deprecated warnings - add note about clicking on results Jordan Sissel 2010-10-29 16:18:49 +00:00
  • 4b9d940341 - Escpae results Jordan Sissel 2010-10-29 16:17:28 +00:00
  • 73714d66d6 - Add javascript I forgot earlier. Jordan Sissel 2010-10-28 19:48:24 +00:00
  • d2f8cadc50 - add Logger support Jordan Sissel 2010-10-28 19:05:14 +00:00
  • 2ac9f69c55 - Update to work with latest code Jordan Sissel 2010-10-28 11:00:27 +00:00
  • 2053ceabcb - revert :port addition Jordan Sissel 2010-10-28 10:59:08 +00:00
  • b8358e6004 - Add '_' to valid characters in URI paths and query parameters Jordan Sissel 2010-10-28 10:49:44 +00:00
  • fb72212e36 - Improve logging. If the logger level == Logger::DEBUG, then we'll log the caller file, line number, and method. Example: Jordan Sissel 2010-10-28 10:39:13 +00:00
  • b711b9d870 - Log when we skip something Jordan Sissel 2010-10-28 09:23:28 +00:00
  • 953fadeb80 - Import from grok svn r3079 (all pattern tests passing) Jordan Sissel 2010-10-28 09:20:07 +00:00
  • 4a89da7c1c - Refine LogStash::Event to have less Hash feel. - Add logging to filter/grok - Make LogStash::Logger::Formatter only use awesome_inspect for non-strings Jordan Sissel 2010-10-28 08:29:34 +00:00
  • 8cb4676420 - Start using our own logger - Add LogStash::Time::to_iso8601 to convert DateTime objects to iso8601 format. - Add a 'date' filter for using a specific field into the real timestamp for the event. Example filters: - date: linux-syslog: date: %b %e %H:%M:%S apache-access: timestamp: "%d/%b/%Y:%H:%M:%S %Z" Syntax is: <tag>: <field>: <format> Supported format is 'strftime' Jordan Sissel 2010-10-28 06:28:32 +00:00
  • bd65c610b4 - fixa logic for setting debug level - start working on our own Logger subclass so we can log objects if necessary Jordan Sissel 2010-10-27 09:03:33 +00:00
  • 62af366598 - remove unused Jordan Sissel 2010-10-27 02:50:35 +00:00
  • 7e9bedea32 - add logging Jordan Sissel 2010-10-27 02:50:22 +00:00
  • 64c072be20 - Add logging Jordan Sissel 2010-10-27 02:47:07 +00:00
  • 7821dd483b - update install notes Jordan Sissel 2010-10-27 02:46:53 +00:00
  • b9592108b2 mark bin/logstash as executable Pete Fritchman 2010-10-26 09:40:37 +00:00
  • 7d1fb812b7 - Add standalone Jordan Sissel 2010-10-26 09:32:29 +00:00
  • 0b44e65eaf - should merge arrays, not append Jordan Sissel 2010-10-26 09:22:47 +00:00
  • 85d90ba6c4 - move grok parsed fields into a separate 'fields' piece Jordan Sissel 2010-10-26 09:02:59 +00:00
  • 7096ab9ebd - Use ajax calls to update search - Make search interactive Jordan Sissel 2010-10-26 06:41:49 +00:00
  • 8fd6ca9422 - Start using haml. Jordan Sissel 2010-10-25 09:53:11 +00:00
  • 49af98b731 - Pad time fraction so it's sortable. - Use strings, not symbols since symbols aren't valid in json (and become strings anyway) Jordan Sissel 2010-10-25 09:52:35 +00:00
  • 983ded0aff - Start working on a new web interface. Previous one was merb, new one is async_sinatra. Initially supporting only ElasticSearch as a backend, will add more eventually. Jordan Sissel 2010-10-24 10:00:42 +00:00
  • b1d2e5e9b1 - Add elasticsearch indexing Jordan Sissel 2010-10-24 04:40:07 +00:00
  • 46c676e7dc - Add readme Jordan Sissel 2010-10-21 20:16:21 +00:00
  • 251e170518 - Add code/script for the logstash websockets screencast. Uses xdotool to type and do other actions on my behalf. Jordan Sissel 2010-10-21 11:14:25 +00:00
  • 6c1ff2abc3 - Hack our hostname into the file url for easy "source" attribution Jordan Sissel 2010-10-18 09:37:51 +00:00
  • 240ff18dce - disable durable for now Jordan Sissel 2010-10-18 08:17:58 +00:00
  • 0acb1d13f3 - Add gemspec Jordan Sissel 2010-10-18 08:05:27 +00:00
  • 694aae5088 - Update packaging - Split config files Jordan Sissel 2010-10-18 07:27:03 +00:00
  • b13f9bf8cb - Add logstash.rb Jordan Sissel 2010-10-18 07:21:15 +00:00
  • d5d532cbc6 - Move everything into the logstash subdir Jordan Sissel 2010-10-18 07:19:37 +00:00
  • ae50ca1d82 - Move LogStash::Components::Agent to LogStash::Agent Jordan Sissel 2010-10-18 00:26:02 +00:00
  • 4c38c9d8fe - Add MongoDB output Jordan Sissel 2010-10-18 00:07:36 +00:00
  • 6805a6ef03 - add Event#to_hash Jordan Sissel 2010-10-18 00:07:21 +00:00
  • 75dd74d398 - remove debug output Jordan Sissel 2010-10-18 00:06:44 +00:00
  • af762f69bb testing websockets Jordan Sissel 2010-10-17 23:24:48 +00:00
  • 0043302675 - Add websocket support (not yet tested) Jordan Sissel 2010-10-17 18:38:21 +00:00
  • e79ae10abe Jordan Sissel 2010-10-17 10:52:36 +00:00
  • d0c4ccbfdc - Enhance the test example to show the ease of reading from local files or simply reading parsed messages from an AMQP topic with the same code and only the configuration changed. Jordan Sissel 2010-10-17 10:49:58 +00:00
  • cf06e05019 Jordan Sissel 2010-10-17 10:36:59 +00:00
  • c6a9bce4fc - Update TODO Jordan Sissel 2010-10-17 10:36:01 +00:00
  • 141a4c64ed - purge old stuff Jordan Sissel 2010-10-17 10:32:13 +00:00
  • 9f3114c883 - Purge old web ui Jordan Sissel 2010-10-17 10:31:26 +00:00
  • 4684f65995 - Purge post-refactor. The older stuff can be found in /tags/pre-agent-refactor Jordan Sissel 2010-10-17 10:30:22 +00:00
  • 791f15f633 - Add the new refactored agent model. An agent can read from any input, apply filters, and pass to any output. * Inputs are files, amqp, stomp, http server, syslog server, etc. * Outputs are similar. * Filters are for manipulating events (parsing, adding data, trimming private data, etc) Inputs so far: amqp, file. Outputs so far: amqp, stdout. Filters so far: grok (pattern discovery only) Jordan Sissel 2010-10-17 10:27:38 +00:00
  • 1575edcdb1 - Add time wrapper for iso8601 support Jordan Sissel 2010-10-17 07:27:24 +00:00
  • d9c8381754 - Add base for unified invocation binary Jordan Sissel 2010-10-17 07:13:38 +00:00
  • 7a6808418f - Don't use File::Tail anymore Jordan Sissel 2010-10-17 06:57:01 +00:00
  • 524194d5de Jordan Sissel 2010-10-17 03:20:05 +00:00
  • c4c880f51d Jordan Sissel 2010-10-17 03:19:57 +00:00
  • d3280be19b - start working on splitting things in to components. I want it to be possible to run a single agent that does log reading, parsing, and indexing locally. Jordan Sissel 2010-10-17 03:19:45 +00:00
  • ae21f3d7c0 - Fix eventmachine-tail usage to match 0.2.x api Jordan Sissel 2010-05-18 08:11:33 +00:00
  • 4aea3bddb3 - Sync grok-patterns from grok svn HEAD. Jordan Sissel 2010-05-18 08:11:06 +00:00
  • c1ecbff37c - If no timezone is in the date format, assume localtime and move to UTC time. Jordan Sissel 2010-05-18 08:10:46 +00:00
  • 6b48e99eaa - Add search form partial Jordan Sissel 2010-05-04 15:53:27 +00:00
  • 58dc837a14 - Add a title to the graph showing the range (time period) - Add a vertical dash on the graph indicating the current time - Make / (index) show a better view of the search form Jordan Sissel 2010-05-04 03:35:33 +00:00
  • 5e955a47ba - Fix output showing result range if total results is less than the query 'limit' - Show date stamp of each result. Jordan Sissel 2010-05-03 21:36:07 +00:00
  • 9795bb56b5 - Fix time conversion for the graph - Fix sort (elasticsearch wants an array of fields, not just a string) - Allow tunable graph steps (default 3600 seconds) - Skip empty capture vaules - add elasticsearch_host to BaseConfig Jordan Sissel 2010-05-02 07:47:01 +00:00
  • 5b9e83ce13 - fix version generation (major.svnrev), current major is 0. - Add missing files to spec Jordan Sissel 2010-04-27 08:36:54 +00:00
  • 4c95afa52c - Add files to spec Jordan Sissel 2010-04-27 08:26:31 +00:00
  • b4865b76b0 - Move elasticsearch support into LogStash::Net::Clients::ElasticSearch Jordan Sissel 2010-04-26 09:26:06 +00:00
  • 877e527d91 - Add elasticsearch_host config option - Make LogStash::Log index to elasticsearch Jordan Sissel 2010-04-26 09:24:09 +00:00
  • 2c299fae25 - Fix search's graph Jordan Sissel 2010-04-25 20:56:39 +00:00
  • 1cb56a94d4 - Use elasticsearch again for a indexing backend. - Update logstash web to hit elasticsearch (hardcoded host for now) Jordan Sissel 2010-04-25 09:12:45 +00:00
  • 32ca7fbaea - copy grok patterns again Jordan Sissel 2010-04-14 09:19:21 +00:00
  • 5c8bef85db - just print matching lines Jordan Sissel 2010-04-14 09:06:36 +00:00
  • 418efeb2fd - Try full-text search, limit 10 results Jordan Sissel 2010-04-14 09:03:42 +00:00
  • 5c9922be05 - Add new indexing that uses TokyoCabinet table db for storage. Jordan Sissel 2010-04-14 07:44:15 +00:00
  • cbfa08081d - Copy new patterns from grok Jordan Sissel 2010-04-14 07:42:33 +00:00
  • 6cd4f63cb4 Jordan Sissel 2010-04-13 08:12:04 +00:00
  • 9cc9b2b952 Jordan Sissel 2010-04-13 08:11:57 +00:00
  • 8965fa6c49 - Rewrite Agent for local-running only. The goal here is mainly to make the agent EventMachine-safe. Later I'll pull the index/parse functionality into a module that can be included into the agent for optional local-only or network modes. Jordan Sissel 2010-04-13 07:03:36 +00:00
  • 7f66dcc2ea - use event machine timers Jordan Sissel 2010-04-11 15:51:36 +00:00
  • e020e8487e - use elasticsearch instead of an indexer Jordan Sissel 2010-03-04 06:33:59 +00:00
  • d84f742554 - can't include a class, only a module. Fix later. Jordan Sissel 2010-02-19 07:50:53 +00:00
  • 056a1d6920 - Move to use ElasticSearch's REST api for indexing instead of our own custom mqrpc+ferret instance. Jordan Sissel 2010-02-19 07:47:10 +00:00
  • 44d8eb4143 - Grab the pidfile before we daemonize so we can report pid lock errors promptly. Jordan Sissel 2010-02-19 06:58:47 +00:00
  • f079e8e2be - fix class usage. InvalidArgument isn't valid, we want ArgumentError Jordan Sissel 2010-01-26 09:05:29 +00:00
  • 4b3460491a - s/info/debug/ for "Sending" messages Pete Fritchman 2009-12-16 23:09:00 +00:00
  • 5143d3b85b - bump to 0.3.6 Pete Fritchman 2009-11-30 22:12:23 +00:00
  • 468ae95ca2 - create a new thread for watching logs so we can properly handle globs. We don't just expand the glob on start, we constantly re-scan for any new log files. Pete Fritchman 2009-11-15 05:35:26 +00:00
  • c5448cf479 - detect when an index is corrupt (index.flush throws an exception), log a fatal error, and kill the entire logstashd process. Pete Fritchman 2009-11-08 07:20:38 +00:00