RFC 3986 (the URI specification) describes the , ; and =
characters used for including parameters in path segments.
Typically these are seen only on the final segment, just before
any query parameters, i.e.
http://www.site.com/path1/path2;jsessionid=OI24B9ASD7BSSD
Adding ; and = to the regex, as , is already included
Description :
Usual syslog message :
<85>Jun 14 15:19:47 localhost sudo: root : TTY=pts/1 ; PWD=/opt/logstash ; USER=root ; COMMAND=/bin/bash
Cisco typical message :
<166> Jun 14 15:30:00 10.100.252.52 %ASA-6-302021: Teardown ICMP connection for faddr 10.100.120.120/0 gaddr 10.100.252.1/0 laddr 10.100.252.1/0
----> program name start with a %
Can be reproduced sending a manual syslog message with python script :
import logging
from logging.handlers import SysLogHandler
#message='Jun 14 15:19:47 localhost sudo: root : TTY=pts/1 ; PWD=/opt/logstash ; USER=root ; COMMAND=/bin/bash'
message=' Jun 15 09:47:36 10.100.252.1 %ASA-6-111116: Teardown UDP connection 6201992 for internet:192.168.1.1/1026 to interne:10.100.120.120/427 duration 0:02:04 bytes 588'
logger = logging.getLogger()
logger.setLevel(logging.INFO)
syslog = SysLogHandler(address=('localhost',5544))
#syslog = SysLogHandler(address='/dev/log')
#formatter = logging.Formatter('%(name)s: %(levelname)s %(message)s')
#syslog.setFormatter(formatter)
logger.addHandler(syslog)
logger.warning(message)
Leading to a "NOT SYSLOG" message in the logs and no @fields{} values
With this change the fields are OK and "NOT SYSLOG" message is gone. I still have a "@tags":["_grokparsefailure"], error though...
In some cases, Onigiruma gets confused about negative matches, so
previously a pattern of '%{QS} something', if false match, would
cause Oniguruma to loop frantickly. I haven't yet dug into
the part of Oni that does this, but it's common that some regexp
engines have this behavior. Easy fix moving to non-backtracking
matches..
- Add more debugging to filters/grok to make it easier to see pattern compile problems.
- Add percentage progress to rate.sh (for tracking long imports on logstash)
- Add content-type to the webserver
- Update to web interface to use new @-prefixed notation (@message, etc)
- remove sample agent.redhat.rb; sample config is the same thing
- indexers now broadcast to each other and build up a directory
- indexers respond to a directory request and provide a list of indexers
- move some errors/usage messages to $stderr
- refactor search.rb to query all available indexers
- load all pattern files under patterns/
- grok captures don't include @LINE, so put it there ourselves
- properly filter grok captures per comments
- tear out grok cmdline stuff
- switch to seconds since epoch for @DATE
- add firewalls pattern with initial netscreen session close pattern
