Add setting for dead_letter_queue.max_bytes to allow a user
to set the maximum possible size of a dead letter queue on disk.
Resolves#7633Fixes#7638Fixes#7652
New boolean setting `config.support_escapes` which defaults to false
(the historical behavior). When set to true, the following escapes are
handled:
* backslash doublequote -> doublequote
* backslash quote -> quote
* backslash n -> newline (ascii 10)
* backslash r -> carriage return (ascii 13)
* backslash backslash -> backslash
* backslash t -> tab (ascii 9)
This will solve #1645.
Fixes#7516
This changes the Java-wide from the default of /dev/random. The reason
for this chnage is that /dev/random can block if there is insufficient
entropy available, and this blocking can cause Logstash to stall on
startup for many many minutes.
Fixes#6117.
This scenario (no entropy) seems especially prone to occur on virtual
machines.
Related: https://github.com/jruby/jruby/wiki/Improving-startup-time#ensure-your-system-has-adequate-entropy
On Windows, I was not sure if this would work, but the JDK source tree
says this:
On Windows systems, specifying the URLs "file:/dev/random" or
"file:/dev/urandom" will enable the native Microsoft CryptoAPI seeding
mechanism for SHA1PRNG.
So defaulting to /dev/urandom should still do the right thing on
Windows.
I also checked to see if other unix-likes had /dev/urandom, and
according to Wikipedia[1], Linux, FreeBSD, OpenBSD, OSX, Solaris,
NetBSD, and others all have urandom.
[1] https://en.wikipedia.org/wiki//dev/randomFixes#7469
Modules contain packaged Logstash configuration, Kibana dashboards and other meta files to ease the set up of the Elastic stack for certain use cases or data sources.
* Introduce a DeadLetterQueueFactory
DeadLetterQueueFactory is a static class that keeps
a static collection of DeadLetterQueueWriteManagers per
pipeline that has plugins requesting to use it.
* DeadLetterQueue was added as a first-class field in the execution context that input/filter/output plugins can leverage
In most cases, this will probably work out, but it will need some testing.
JRuby and `logstash.lib.sh` both check first for the presence of `JAVACMD`. If that isn't found, it tries `$JAVA_HOME/bin/java`. If that isn't found, it tries `java` in the system $PATH.
The previous behavior was to manually assign `JAVACMD=/usr/bin/java`. This will still work for most users. Special cases will still need manual configuration, but this will _probably_ work better for most cases in the long run.
Fixes#6919
add queue.max_acked_checkpoint and queue.checkpoint_rate settings
now using checkpoint.max_acks, checkpoint.max_writes and checkpoint.max_interval
rename options
wip rework checkpointing
refactored full acked pages handling on acking and recovery
correclty close queue
proper queue open/recovery
checkpoint dump utility
checkpoint on writes
removed debug code and added missing newline
added better comment on contiguous checkpoints
fix spec for new pipeline setting
sets default to INFO and updates some verbose logging to
more appropriate, less verbose log levels where it makes sense.
Closes#5735.
Closes#5893.
Fixes#5898
This PR does a few modifications for our webserver:
- Add a PortRange setting type.
- allow the --http.port option to accept a port range or a single port, when a range is provided Logstash will incrementally try this list to find a free port for the webserver.
- Logstash will report in the log which port it is using. (INFO LEVEL)
- It refactors a few methods of the webserver.
- It adds test for the binding of the ports.
Notes:
Port range can be defined in the logstash.yml or pass via the command line like this.
`bin/logstash -e 'input { generator {}} output { null {}}' --log.level verbose --http.port 2000-2005`
Fixes#5774
This fix is for packages only (RPM/DEB)
This puts the requisite changes into /etc/logstash/logstash.yml so that starting logstash from the command-line, e.g. `/usr/share/logstash/bin/logstash --path.settings /etc/logstash` will read the pipeline from `/etc/logstash/conf.d` and log to `/var/log/logstash/logstash.log`.
fixes#5379
Comment out tmpdir in jvm.options
The absence of the tmpdir can cause JRuby to fail
Fixes#5383
Update after-install scripts
Use /usr/lib/logstash instead of /opt/logstash
More revisions.
Using `/usr/share/logstash` instead of `/usr/lib/logstash`
Track these new files
More work in progress.
Proper arg parsing obtained
WIP: Use jvm.options like ES
Remove facter dependency
Due to an update from @jordansissel on pleaserun, facter is no longer necessary
Revert Gemfile to proper version
Latest work in progress.
I added a simple test to logstash.lib.sh to catch empty strings in java options.
Added `startup.options`, and touched up `jvm.options`
Edited the package scripts to make things play nice with new paths.
Tweaked `system-install` to allow users to provide their own `startup.options` file.
Patch up merge failure
Changes to work with the new logstash.yml
Add fpm dependency (build fails without it)
Fix missing gems in Gemfile
Don't know how these got lost
Use /etc/logstash for configuration in packages
Update the instructions/documentation for...
...startup.options
Merging proposed changes
Remove mustache and clamp
They're covered by pleaserun anyway
Prune config dir from /usr/share/logstash...
...but only when using RPM/DEB packages
* by default lives in LOGSTASH_HOME/config/logstash.yml
* location can be changed by $LS_SETTINGS_DIR or --path.settings
* overrides defaults of LogStash::Environment, BUT
* CLI flags override the yaml file
* several breaking changes to cli flag names (see config/logstash.yml)
* setting values are now type checked (see Setting and Settings classes)
Fixes#5313
