Commit graph

240 commits

Author SHA1 Message Date
Colin Surprenant
d79e3730fe exclusive file locking class and tests
use new bin/ruby and bundler without :development

refactor to DRY and use expected exception

added original Apache 2.0 license and some cosmetics

exclude bin/lock from packaging

rename variables
2017-01-31 13:47:07 -05:00
Colin Surprenant
ca9ff9172d add bin/ruby command to launch ruby scripts using the logstash launcher
use bin/ruby instead of hardcoded vendorized jruby

without :development

remove symlink check per PR discussiom, will re-introduce in a followup PR
2017-01-30 17:05:23 -05:00
Aaron Mildenstein
35999aeeab Change system-install to allow manual selection
Now you can specify 3 arguments, each dependent on the previous:

1. full path to the `startup.options` file
2. the startup variant (sysv, upstart, systemd)
3. the version of the startup variant (usually unnecessary, can safely be left blank)

Present behavior is preserved.  If `system-install` is called with no arguments, it autodetects and installs the detected variant.

Fixes #6338
2016-12-09 12:57:16 -05:00
Aaron Mildenstein
60b6a9f72e Have logstash.lib.sh find jvm.options
This will work if `--path.settings` is properly used on the command line

Remove unused count variable

It isn't strictly necessary

Add double-quotes around $@

This should protect against strange spaces

fixes #6379

Fixes #6382
2016-12-09 12:19:22 -05:00
Bartłomiej Palmowski
d5d556989d Don't pass jruby opts explicitly
they are already passed via JAVA_OPTS environment variable.

Fixes #5830
2016-12-06 13:24:59 -05:00
Gérald QUINTANA
2bb69743e6 Fix Heap Dump Path on Windows
Fixes #6336
2016-12-01 23:20:38 -05:00
Shinjifox
8c9b874c98 Fixed typo for logstash cli help
The original shell script throws a error when calling it with "bin/logstash help"
# ./logstash help
ERROR: Unknown command 'help'

See: 'bin/logstash --help'

Fixes #6303
2016-11-29 05:46:42 -05:00
Colin Surprenant
f636a751f8 add support for queue.checkpoint.{acks|writes} settings
add queue.max_acked_checkpoint and queue.checkpoint_rate settings

now using checkpoint.max_acks, checkpoint.max_writes and checkpoint.max_interval

rename options

wip rework checkpointing

refactored full acked pages handling on acking and recovery

correclty close queue

proper queue open/recovery

checkpoint dump utility

checkpoint on writes

removed debug code and added missing newline

added better comment on contiguous checkpoints

fix spec for new pipeline setting
2016-11-22 14:48:58 -05:00
汪啸
1f1bbb1f89 fix wrong logic when no JAVA_OPTS set
Fixes #6034
2016-10-14 17:55:43 -04:00
Joao Duarte
72b277404b make -V/--version fast on *nix 2016-06-21 14:30:15 +01:00
Joao Duarte
bb0694d69a Revert "make -V/--version fast on *nix"
This reverts commit 562754b11a.
2016-06-17 10:00:57 +01:00
Joao Duarte
562754b11a make -V/--version fast on *nix 2016-06-16 12:46:23 +01:00
Aaron Mildenstein
4baf29dc5f Add SINCEDB_DIR env var to logstash.lib.sh
This is to fix the file input

Fixes #5385
2016-05-27 22:08:58 +00:00
Aaron Mildenstein
e40ecbb6b8 Update existing packaging to use /usr/lib/logstash
Update after-install scripts

Use /usr/lib/logstash instead of /opt/logstash

More revisions.

Using `/usr/share/logstash` instead of `/usr/lib/logstash`

Track these new files

More work in progress.

Proper arg parsing obtained

WIP: Use jvm.options like ES

Remove facter dependency

Due to an update from @jordansissel on pleaserun, facter is no longer necessary

Revert Gemfile to proper version

Latest work in progress.

I added a simple test to logstash.lib.sh to catch empty strings in java options.

Added `startup.options`, and touched up `jvm.options`

Edited the package scripts to make things play nice with new paths.

Tweaked `system-install` to allow users to provide their own `startup.options` file.

Patch up merge failure

Changes to work with the new logstash.yml

Add fpm dependency (build fails without it)

Fix missing gems in Gemfile

Don't know how these got lost

Use /etc/logstash for configuration in packages

Update the instructions/documentation for...

...startup.options

Merging proposed changes

Remove mustache and clamp

They're covered by pleaserun anyway

Prune config dir from /usr/share/logstash...

...but only when using RPM/DEB packages

Fixes #5341
2016-05-25 17:25:26 +00:00
Bernd
65b1f991b0 Support ( in path and actually set JAVA_OPTS
The current logic for printing %JAVA_HOME% inside if ( ) block failes if the path contains a `(` which is the case for 32bit java (`C:\Program Files (x86)\java\jdk`)

The current logic for appending to JAVA_OPTS does not work as it will be expanded while interpreting the else block. Use goto instead.

Fixes #5214
2016-05-06 16:52:36 +00:00
Stefan Scherer
a91bf9c40b Use JAVA_EXE as variable for symlink
Signed-off-by: Stefan Scherer <scherer_stefan@icloud.com>

Fixes #4913
2016-04-21 19:46:05 +00:00
Stefan Scherer
25386c488c Show JAVA_HOME retrieved from symlink
Signed-off-by: Stefan Scherer <scherer_stefan@icloud.com>

Fixes #4913
2016-04-21 19:46:04 +00:00
Stefan Scherer
4830399de0 Retrieve JAVA_HOME from windows symlink
Signed-off-by: Stefan Scherer <scherer_stefan@icloud.com>

Fixes #4913
2016-04-21 19:46:04 +00:00
Suyog Rao
4b07413cd9 Changes after review from @purbon
Fixes #4874
2016-03-24 00:23:15 +00:00
Suyog Rao
683302e26d Rename bin/plugin to bin/logstash-plugin
This adds support for packs and fixes the problem where bin/plugin
pollutes the PATH namespace

Fixes #4871

Fixes #4874
2016-03-24 00:23:15 +00:00
Todd Bowles
62ab6f0e40 Improved the way in which JAVA_OPTS and LS_JAVA_OPTS work on windows by making it consistent with the way in which they are handled in Linux
Fixes #4758
2016-03-11 20:27:29 +00:00
Siddhartha Sahu
02f5b54c46 Add ability to configure custom gc log file
This adds ability to add gc log file to the custom $LS_LOG_DIR path

Fixes #4718
2016-02-26 19:30:11 +00:00
Suyog Rao
c136d58d32 Set LS_HEAP_SIZE correctly to 1g
Previously we had set heap size to 1g in a few places, but not everywhere

Fixes #4550
2016-01-22 17:17:34 +00:00
Aaron Mildenstein
68bdde886a Allow Logstash to be launched from a symlink
These changes allow Logstash to be launched from a symlink _only_ when the `readlink`, or suitable version of `stat` binaries are present. An error message will result if Logstash is launched from a symlink and the path cannot be discovered by one of these two methods.  The message will advise the user to use the full path instead of a symlink to launch Logstash.

fixes #4291
2015-12-04 15:25:54 -08:00
Joao Duarte
44edf5bd00 allow logstash to run in a dir with spaces
because HeapDumpPath is set, if it contains spaces the jruby_opts
function will create two strings appended by -J
even if the list of jruby opts is constructed using replacement
shell script will not escape the path correct to the heap dump path
option needs to be added as late as possible

this is a *nix fix only, windows is not affected

Fixes #4303
2015-12-04 21:27:57 +00:00
Joao Duarte
00a99c19e5 refactor concerns of runner, agent and pipeline
* move cli argument handling from agent to runner
* add a short-help message
* add interactive shell option
* log fatal messages to terminal when logging to file
* change docs:generate task to use bundle exec

Fixes #3872
2015-11-25 12:11:36 +00:00
Suyog Rao
ba88ff7d78 Uncomment heapdump location
Fixes #4205
2015-11-17 15:53:41 +00:00
Suyog Rao
d124558d79 Fix heapdump path for Windows
Fixes #4205
2015-11-17 15:53:41 +00:00
Suyog Rao
f1775d0b7e Update comment about location
Fixes #4205
2015-11-17 15:53:41 +00:00
Suyog Rao
d69cae23e6 Add option for JVM to dump its heap on OutOfMemory
Fixes #4192

Fixes #4205
2015-11-17 15:53:41 +00:00
Pier-Hugues Pellerin
1045ec8602 Change how we set the max heap size
This PR change the name of the variable to set to change the maximum
heap size on window to `LS_HEAP_SIZE` this replace the LS_MIN_MEM and
LS_MAX_MEM option.

Also We change the default size of the heap from 500m to 1g

Fixes #4085
2015-10-30 19:50:51 +00:00
Colin Surprenant
54fb7d6c78 fix bin/bundle command 2015-10-21 10:45:37 -04:00
Manus Freedom
456c726890 Fix path with space
Fixes #3372
2015-06-10 06:15:52 +00:00
Manus Freedom
cf959c7182 Revert "use windows short names => no issue with space in dir" It's a very bad idea to use short name, because you can disable short name on windows.
This reverts commit 6cb438d7fc.

Fixes #3372
2015-06-10 06:15:51 +00:00
wiibaa
6cb438d7fc use windows short names => no issue with space in dir
Fixes #3252
2015-05-19 08:16:07 +00:00
Colin Surprenant
a939a2e6ca add file content :P
Fixes #3231
2015-05-12 18:52:18 +00:00
Colin Surprenant
036db56fea windows bin/rspec.bat
Fixes #3231
2015-05-12 18:52:17 +00:00
Colin Surprenant
933709fba4 unset CDPATH 2015-05-11 16:17:15 -04:00
Colin Surprenant
48bcc5b74d cleanup dependencies and rely on logstash-core gem
remove unused load path

use either local core lib dir or logstash-core gem

include spec in logstash-core

do not include lib, spec and locales dirs in package root

environment bootstrapping in lib/bootstrap.rb

added comments

move pluginmanager out of logstash-core

kafla installation solved by pluginmanager refactor

refactor bootstrap code

refactor shell scripts to simplify and DRY, indirect rspec command to our ruby launcher

add bin/rspec

cut by half the bin/plugin and bin/rspec startup time

fix drip context

use printf instead of tr

updated Windows bin scripts

missing gemspec in gemspec

use gem instead of gemspec so our plugin manager can correctly install logstash-core

generate packages including locally built logstash-core gem

move jenkins developmnent dependencies into Gemfile, they do not belong in logstash-core

path leftover

clean help agent help usage message and remore rspec in available command

comments cosmetics

update Bundler dependency, all recent testing have been made with 1.9.4

updated .lock file with regard to updated Gemfile

cleanup gemspec, Gemfile and regen .lock file

added progress output

avoid dual require on version

closes #3096
2015-05-04 18:17:22 -04:00
Pere Urbon-Bayes
0b72d06de8 add an rspec shim to complement the old bin/logstash rspec way of running specs in logstash
simplify the rspec shim codebase

add bin/rspec to the exclude list for packaging

removed leftover

Fixes #3047
2015-04-20 15:47:46 +00:00
Colin Surprenant
bde2691239 bin/bundle shim
use  instead for $:

exclude bin/bundle from packages

added comment
2015-04-20 17:07:52 +02:00
Pere Urbon-Bayes
193765ced2 adapt the lib.sh file to the use case proposed LS_JAVA_OPTS add to defualts while JAVA_OPTS override
updated the packaged to enable JAVA_OPTS to override the defaults

ammended JAVA_OPTS being exported

ammend warning wording

ammend the last commit

fix wording again

Fixes #2942
2015-04-17 13:49:19 +00:00
Pere Urbon-Bayes
cbaadb16f2 make the LS_JAVA_OPTS environment variable available to the client side, so the user is not force to use the default ones and they can rewrite it
remove stale code

cleanup the bash conditions for when a variable is defined or not

make sure LS_JAVA_OPTS override JAVA_OPTS when need

ammend unused var

rollback to previous use case of appending vars to JAVA_OPTS in the case of having LS_JAVA_OPTS

Fixes #2942
2015-04-17 13:49:19 +00:00
Aaron Mildenstein
3986158bac First pass at cleaning out logstash web
All internal references, flags, kibana, vendor, everything must go!

If it referenced kibana or logstash-web, it is deprecated, and removed!

Operators are standing by...

Fixes #2661
2015-02-20 22:53:18 +00:00
Pier-Hugues Pellerin
9e7d6204a5 plugins/bundler refactor
squashed 113 commits by @ph, @colinsurprenant, @jordansissel, @purbon, @jsvd

cleaning the environment.rb

initial work

remove multiples polyglot

force to load rubygems

notice

point to one config

update lock

bootstrap will package the standard application until we have the plugin manager on top of the gemfile

add the gem path to find bundler

fixing dependencies

add gemfiles and bundle

comment platform requiremments in the gemfile

Set `Gem.paths = ...` to the GEM_HOME

Without this, `bin/logstash` will error:

    LoadError: no such file to load -- bundler
         require at org/jruby/RubyKernel.java:1071
         require at C:/Users/jls/Documents/GitHub/logstash/vendor/jruby/lib/ruby/shared/rubygems/core_ext/kernel_require
.rb:55
         set_gem_paths! at C:/Users/jls/Documents/GitHub/logstash/lib/logstash/environment.rb:64
         (root) at C:\Users\jls\Documents\GitHub\logstash\lib\logstash\runner.rb:8

Patch JRuby's jar_installer.rb with a newer version

jar-dependencies is shipped by default with JRuby. However, the version
it ships with has a bug, and despite upgrading jar-dependencies to
0.1.7, which has the fix, JRuby continues to load the "shipped" version
which has a bug.

This patch will patch JRuby by copying in our own jar_installer.rb from
jar-dependencies master, This patching occurs while unpacking JRuby just
after `vendor:jruby`.

The jar_installer.rb is MIT-licensed and no changes were made.

Add `test:plugins` to test all installed plugins

Tell rspec to do the wildcards instead of trying to make the shell do it.

This fixes test runs on Windows (which doesn't do wildcards like this).
The solution was to tell RSpec to look for all files matching a certain path pattern
within the GEM_HOME.

do not run bundle install twice for test:plugins

Gemfile manipulation classes

Gemfile now in root

install POC with new Gemfile classes

specs for the Gemfile classes

header, some bug fixes

spec header

WIP pluginmanager refactor

generated format

updated .lock

unused file

removed --proxy option

fail elegantly

initial support for update

refactor invoke_bundler, debug output

removed old .lock file

no plugin Gemfile

use clamp error handling

reenable list, some cleanups

support for more finders

refactored using new finders

refactor using is_installed_plugin? and some cleanups

verify valid plugin and update all now only update plugins not all gems

support multiple plugins argument, --force option to bypass validation

list only explicitly installed plugins by default, --all option to include dependencies

support multiple plugins argument

renamed all_installed_plugins to all_installed_plugins_gem_specs and return specs, added comments

remove then

update_gem bug, comments

moved gems as development dependencies

updated .lock file

support for :without

--development option, better output feedback, cleanups

abort when no plugins

cosmetic

param fix, cleanups

removed deprecated .lock files

remove deprecated bundler_install_command and struggling with set_gem_paths

refactored all bundler stuff into LogStash::Bundler

rakelib refactor

refactored bundler code

Bundler.real_fuckin_reset

real_fuckin_reset is now reset_settings, removed unnecessary resetings, cleanups

duplicate plugins requirements merging

cleanups

added vendor:rake, cleanups

broke test tasks into 2 steps, first invoke install then test

rake task name fix

proper error exit code

Freezing the jar-dependencies, ruby-maven, tools

set gem paths on error

renamed TEST_PLUGINS to CORE_PLUGINS, added comments

replace test by core, cleanups

DRYied test tasks

missing gemspec

fixed output

correct image build for artifacts

removed deprecated Gemfiles

fix update with multiple plugins arguments

fix update with multiple plugins arguments

remove deprecated code

implicit begin, explicit block

spec for LogStash::Bundler

remove explit require of rubygems

renamed Environment.set_gem_paths! to bundler_setup!

rename Dsl to DSL

respect to_s contract and return empty string not nil

rename DSL.parse gemfile argument to gemfile_content

better error message

renamed --force to --[no-]verify

refactored exit error handling

use explicit array items in iterators

fix --development switch validation, cleaned bundler_options hash init

renamed is_xxx? methods to xxx?

use direct method call on collection

removed double negative, typo

move maven tools monkey patch into maven_tools_patch.rb

renamed build-image to prepare

fix forgotten argument name change

namespace cleanups

spec files glob pattern

cleanup

cleanup

cleanup

Fix the file-dependencies version to 0.1.5 to fix path issues on windows

monkeypatch Rubygems to silence ffi warning

list all plugin gems by default, new options

adding rake task to get faster feedback loop on windows

refactor bat scripts

Use an unreleased filedependencies gem and skip installing drupal

bump file-dependencies

use the released gem of file-dependencies

Add a set of scripts that simplify:   * The necessary setup for the CI test to be executed.   * The actuall CI test execution, having a different scripts for each     different workflow. This makes the test execution simpler and standard, so changes in internal task don't need big changes in the CI config. It also documents and makes easy for developers to run the tests.

Fixes #2640

refactored test to not depend on STDOUT

create a .rspec file to pass some options used while the code is in the testing environment

add coveralls and simplecov as a development dependencies so we can run coverage tests

remove unnecessary .bundle directory from the artifact

In a release, the .bundle/config has a BUNDLE_PATH setting that can
interfere by misleading the plugin manager into not seeing the
vendor/bundle path.

Fixes #2647

Fixes #2475
2015-02-20 17:31:43 +00:00
Jordan Sissel
f0da80f090 Add bin/plugin.bat which just invokes bin/logstash plugin ...
Fixes #2287

Fixes #2451
2015-01-28 00:02:50 +00:00
Pere Urbon-Bayes
05a097f853 Make sure the vendorized jruby only have 1.9 libs, so no extra space is used. Also make sure the jruby is run in 1.9 mode.
Fixes #2285
2015-01-05 15:50:15 +00:00
wiibaa
98abbafb3d allow implicit agent using directly a flag in cli
Fixes #1296
2014-12-23 12:43:19 +00:00
wiibaa
395cb7563b Update for windows to execute rake bootstrap and bin\logstash.bat
Fixes #2085
2014-12-22 21:46:25 +00:00
Suyog Rao
0956e2a3ae Use exec instead of sourcing
Fixes #2017
2014-11-05 02:45:34 +00:00