github-mirrors/logstash
1
0
Fork 0
mirror of https://github.com/elastic/logstash.git synced 2025-04-21 05:08:07 -04:00
Code Issues Projects Releases Packages Wiki Activity
11063 commits 255 branches 379 tags 173 MiB
Commit graph

3734 commits

Author SHA1 Message Date
Jordan Sissel
3c9c38e55e - add proper setters 2010-11-11 08:06:03 +00:00
Jordan Sissel
955773f4ff - make grokdiscovery work 2010-11-09 10:19:38 +00:00
Jordan Sissel
ca568c0437 - Add grokdiscovery 2010-11-09 10:11:28 +00:00
Jordan Sissel
37488a44fc - use DateTime.parse to ensure any ISO8601 format dates rae in our common 
format. This may not be necessary in the long run.
 2010-11-09 09:58:28 +00:00
Jordan Sissel
5a8a350ef3 - Degrade-without-dying if we don't have awesome_print 2010-11-09 09:50:29 +00:00
Jordan Sissel
a579aea186 - Allow easy specification of ISO8601 format 2010-11-09 09:49:55 +00:00
Jordan Sissel
2953457843 - Add all patterns found 2010-11-09 09:27:57 +00:00
Pete Fritchman
ec2ae60d83 add support for streaming logs to elasticsearch via a "rabbitmq river". 
we have to PUT a json config to the elasticsearch HTTP API to tell it
where to look, then we use our amqp output to send bulk-data style
index messages to elasticsearch. The ES side does bulk-queueing for us.
 2010-11-01 01:46:20 +00:00
Pete Fritchman
b250a0218c move CGI include around to proper place 2010-11-01 01:40:29 +00:00
Pete Fritchman
84ef233bca - parse query string on output URLs for additional options 
- amqp now properly supports "queue" type (and in general, works now)
- amqp supports durable queues (?durable=1)
- added amqp#receive_raw for publishing a string directly
 2010-11-01 01:35:52 +00:00
Jordan Sissel
57f480adeb - Make TZ in DATESTAMP_OTHER optional 
- Add more debugging to filters/grok to make it easier to see pattern compile problems.
- Add percentage progress to rate.sh (for tracking long imports on logstash)
- Add content-type to the webserver
- Update to web interface to use new @-prefixed notation (@message, etc)
 2010-10-31 08:59:28 +00:00
Jordan Sissel
b3e67888a1 2010-10-31 08:52:41 +00:00
Jordan Sissel
8ef912a624 - Only push events to websockets when we have subscribers 2010-10-31 08:28:55 +00:00
Pete Fritchman
8462bd05fa introduce log types, in addition to tags 2010-10-31 06:01:46 +00:00
Pete Fritchman
8d6419e191 s/throw/raise/ 2010-10-31 04:50:19 +00:00
Pete Fritchman
f7ad0a0a9a create base classes for inputs/outputs/filters 2010-10-31 04:46:12 +00:00
Jordan Sissel
2aba74e2f4 - metadata uses '@' prefix notation. (@timestamp, @source, etc) 2010-10-31 02:22:12 +00:00
Jordan Sissel
fb72212e36 - Improve logging. If the logger level == Logger::DEBUG, then we'll log the 
caller file, line number, and method. Example:

  With 'logger.level == Logger::DEBUG'
  I, [2010-10-28T03:38:23.984737 #25011]  INFO -- logtest.rb:9#foo: Hello
     ^ includes source file, line, and method. Source file gets trimmed
     if it's in $: (RUBYLIB path)
                                                  
  With 'logger.level != Logger::DEBUG'
  I, [2010-10-28T03:37:42.235899 #24998]  INFO -- logtest.rb: Hello
                                                     ^ progname, aka $0
 2010-10-28 10:39:13 +00:00
Jordan Sissel
b711b9d870 - Log when we skip something 2010-10-28 09:23:28 +00:00
Jordan Sissel
4a89da7c1c - Refine LogStash::Event to have less Hash feel. 
- Add logging to filter/grok
- Make LogStash::Logger::Formatter only use awesome_inspect for non-strings
 2010-10-28 08:29:34 +00:00
Jordan Sissel
8cb4676420 - Start using our own logger 
- Add LogStash::Time::to_iso8601 to convert DateTime objects to iso8601 format.
- Add a 'date' filter for using a specific field into the real timestamp for
  the event. Example
    filters:
    - date:
        linux-syslog:
          date: %b %e %H:%M:%S
        apache-access:
          timestamp: "%d/%b/%Y:%H:%M:%S %Z"
  Syntax is:
    <tag>:
       <field>: <format>
  Supported format is 'strftime'
 2010-10-28 06:28:32 +00:00
Jordan Sissel
bd65c610b4 - fixa logic for setting debug level 
- start working on our own Logger subclass so we can log objects if necessary
 2010-10-27 09:03:33 +00:00
Jordan Sissel
62af366598 - remove unused 2010-10-27 02:50:35 +00:00
Jordan Sissel
7e9bedea32 - add logging 2010-10-27 02:50:22 +00:00
Jordan Sissel
64c072be20 - Add logging 2010-10-27 02:47:07 +00:00
Jordan Sissel
0b44e65eaf - should merge arrays, not append 2010-10-26 09:22:47 +00:00
Jordan Sissel
85d90ba6c4 - move grok parsed fields into a separate 'fields' piece 2010-10-26 09:02:59 +00:00
Jordan Sissel
49af98b731 - Pad time fraction so it's sortable. 
- Use strings, not symbols since symbols aren't valid in json (and become
  strings anyway)
 2010-10-25 09:52:35 +00:00
Jordan Sissel
b1d2e5e9b1 - Add elasticsearch indexing 2010-10-24 04:40:07 +00:00
Jordan Sissel
6c1ff2abc3 - Hack our hostname into the file url for easy "source" attribution 2010-10-18 09:37:51 +00:00
Jordan Sissel
240ff18dce - disable durable for now 2010-10-18 08:17:58 +00:00
Jordan Sissel
b13f9bf8cb - Add logstash.rb 2010-10-18 07:21:15 +00:00
Jordan Sissel
d5d532cbc6 - Move everything into the logstash subdir 2010-10-18 07:19:37 +00:00
Jordan Sissel
ae50ca1d82 - Move LogStash::Components::Agent to LogStash::Agent 2010-10-18 00:26:02 +00:00
Jordan Sissel
4c38c9d8fe - Add MongoDB output 2010-10-18 00:07:36 +00:00
Jordan Sissel
6805a6ef03 - add Event#to_hash 2010-10-18 00:07:21 +00:00
Jordan Sissel
75dd74d398 - remove debug output 2010-10-18 00:06:44 +00:00
Jordan Sissel
0043302675 - Add websocket support (not yet tested) 2010-10-17 18:38:21 +00:00
Jordan Sissel
d0c4ccbfdc - Enhance the test example to show the ease of reading from local files 
or simply reading parsed messages from an AMQP topic with the same code
  and only the configuration changed.
 2010-10-17 10:49:58 +00:00
Jordan Sissel
141a4c64ed - purge old stuff 2010-10-17 10:32:13 +00:00
Jordan Sissel
4684f65995 - Purge post-refactor. The older stuff can be found in /tags/pre-agent-refactor 2010-10-17 10:30:22 +00:00
Jordan Sissel
791f15f633 - Add the new refactored agent model. 
An agent can read from any input, apply filters, and pass to any output.
    * Inputs are files, amqp, stomp, http server, syslog server, etc.
    * Outputs are similar.
    * Filters are for manipulating events (parsing, adding data, trimming
      private data, etc)
  Inputs so far: amqp, file.
  Outputs so far: amqp, stdout.
  Filters so far: grok (pattern discovery only)

  A sample custom agent is in examples/test.rb
 2010-10-17 10:27:38 +00:00
Jordan Sissel
1575edcdb1 - Add time wrapper for iso8601 support 2010-10-17 07:27:24 +00:00
Jordan Sissel
7a6808418f - Don't use File::Tail anymore 2010-10-17 06:57:01 +00:00
Jordan Sissel
524194d5de 2010-10-17 03:20:05 +00:00
Jordan Sissel
c4c880f51d 2010-10-17 03:19:57 +00:00
Jordan Sissel
d3280be19b - start working on splitting things in to components. I want it to be possible 
to run a single agent that does log reading, parsing, and indexing locally.
 2010-10-17 03:19:45 +00:00
Jordan Sissel
ae21f3d7c0 - Fix eventmachine-tail usage to match 0.2.x api 2010-05-18 08:11:33 +00:00
Jordan Sissel
c1ecbff37c - If no timezone is in the date format, assume localtime and move to UTC time. 2010-05-18 08:10:46 +00:00
Jordan Sissel
9795bb56b5 - Fix time conversion for the graph 
- Fix sort (elasticsearch wants an array of fields, not just a string)
- Allow tunable graph steps (default 3600 seconds)
- Skip empty capture vaules
- add elasticsearch_host to BaseConfig
 2010-05-02 07:47:01 +00:00