Commit graph

180 commits

Author SHA1 Message Date
Suyog Rao
62f56ec274 Fix issue when building regular tar
Thanks to @ph who caught this issue in review
https://github.com/elastic/logstash/pull/4435#discussion_r49224181

Fixes #4440
2016-01-09 02:27:28 +00:00
Suyog Rao
6e94d0c68b Add rake tasks to create uber packages
Add rake tasks to create zip and tar.gz for LS package with all plugins
installed from losgtash-repo

Fixes #4435
2016-01-08 23:08:21 +00:00
Andrew Cholakian
c2715d0ded Add http poller to default plugins list
Fixes #4409
2016-01-03 23:06:44 +00:00
Vincent Bernat
4bc5261895 pkg: ensure dev/beta/rc are ordered before a regular version
```ruby
class DpkgVersion
  include Comparable
  attr :version
  def <=>(anOther)
    return -1 if system("dpkg --compare-versions #{version} lt #{anOther.version}")
    return 1 if system("dpkg --compare-versions #{version} gt #{anOther.version}")
    return 0
  end
  def initialize(version)
    @version = version
  end
  def to_s
    @version
  end
end

versions="1.5.1 1.5.0 1.5.0.rc3 1.5.0.beta2 1.5.0.alpha3".split
versions=versions.map { |x| DpkgVersion.new(x.gsub(/\.([[:alpha:]])/, '~\1')) }
versions=versions.sort
puts versions
```

This outputs:

```
1.5.0~alpha3
1.5.0~beta2
1.5.0~rc3
1.5.0
1.5.1
```

Fixes #2461
2015-12-11 20:17:33 +00:00
Joao Duarte
00a99c19e5 refactor concerns of runner, agent and pipeline
* move cli argument handling from agent to runner
* add a short-help message
* add interactive shell option
* log fatal messages to terminal when logging to file
* change docs:generate task to use bundle exec

Fixes #3872
2015-11-25 12:11:36 +00:00
Pere Urbon-Bayes
245319e9b8 update jruby to last 1.7.23 release
Fixes #4260
2015-11-25 10:04:04 +00:00
Pere Urbon-Bayes
2195f145b2 This adds a feature to let users dump all their installed plugins,
including dependencies, and reuse them in an offline installation by
providing an package for it. It adds two important commands to the
plugin manager, the pack and upack, to handle package creation and
installation and adds the --local flag to install and update to pull
plugins from the installed local packages.

Former commits:

add a task to create a bundle, under vendor/cache of the installed gems + their dependencies, this can be used later on to be installed offline

add an option to pass the --local flag to the bin/plugin update task, so it fetch plugins from a local cache, by default under vendor/cache

rename package:build to package:build-cache as is more meaningfull

add a --local flag to bin/plugin install to users can install plugins from the local cache, under the default directory of vendor/cache

add a plugin manager command to build the local cache of installed plugins using bundler package alike command

It adds code for compressing and extracting in zip and tar formats to
the utils module of logstash. The zip module is only jruby dependant as
it uses functions provided by java.
There is also code add in the plugin manager package command to handle
compression of plugins dumping.

Cleanup the custom java code to compress and extract zip files as it has
been known that using rubyzip is good and it has a more ruby like
features so the code is more clean and portable.

set of smallish improvement requested during review

added several options to handle situation when the package of plugins we want to generate is already there

clean up old code

applyed feedback from review, mostly changed in documentating behaviour plus better wording for messages

relocate the Environment.windows? check to the bootstrap side as it's used also in the plugin manager

add an unpack bin/plugin command so users can install packages of plugins throw the plugin manager

document override behaviour in the compress helpers (for zip and tar) plus add a fix for the tar extract when reading entries

made the unpack filename parameter a required one

add a force option to the bin/plugin unpack command

add a notice to that if using a local cache the --local flag should be passed with

Code cleanup and refactor introduced during review

add two wording suggestions comming from review

ammend more wording

skip the major version validation as in situation where there is no internet connection like when using the local vendor/cache to do the update

move compress to the bootstrap environment as being used in the plugin manager means not being loaded with logstash-core

Bring pack cached gems in the vendor directory so they can be used for bundler when packaging dependencies

Revert "Bring pack cached gems in the vendor directory so they can be used for bundler when packaging dependencies"

This reverts commit a9d7f46649932b06efaafebdd0eed2b4c63c2235.

patch the Bundler::Source::Rubygems to fetch only gems from a remote  source

small changes in several parts of the plugin manager and the creation of a common pack command with shared code

change compress to read in chuncks

fix wrong var name in the bootstrap compress utils module

fix namespacing conflicts

add basic test for the compress utility module

ammend more namespace issues

add a comment to the rubygems mockey patch to rebuild the gem cache

apply cosmetic changes

make the compress module raise CompressError

remove vendor_path and pattern_path form environment as they where mix up during rebase

change the bin/pack force option to be force-delete-cache

rename force_delete_cache to overwrite

change force for overwrite in tha pack command

remove the override option for bin/plugin unpack

revert Gemfile and Genfile.lock wrong committed
2015-11-18 12:03:51 +01:00
Colin Surprenant
1a32c14eac fix version path, relates to #4123 2015-11-17 01:13:11 -05:00
Colin Surprenant
d74d41cb30 logstash-core & logstash-core-event extraction to support logstash-core-event-java impl, relates to #4191
fixed timezone issue

extracted logstash-core and reorganized specs

extracted logstash-core-event

extract java Event into logstash-core-event-java in a proper gem

remove obsolete jruby_event bootstrapping

fix require path

add java code bootstrap

use logstash-core-event/logstash-core-event.rb

remove obsolete files

basic instructions

LogStash::Json need to be initialized from event

update jruby and gradle versions

update compile:logstash-core-event-java rake task

WIP tasks refactor

fix gem.files

skip test if class is not defined

fix gem related tasks for new structure

add gem spec dirs in core tests

bootstrap java implementation when requiring timestamp

new Cloner class and Event clone impl

fix array fields assignments, see #4140

don't rely on json implementation ordering

fix skipped last iterpolation char

remove implementation specific unnecessary check

also require ruby classes

define error class in ruby

raise exception on invalid format

remove implementation specific tests and extract and put logger related test in pending

missing bits for having all core timestamp specs pass

run all core specs

remove leftover

comment regex

missing encoding header

revert to logstash-core-event by default

finished proper gemification

useless require

dynamically pick specs depending on logstash-core-event-* implementation

logstash root package version

missing file for proper gemification

do not build java event by default

always check for root logstash lib dir

fix concurrent-ruby version confict

fix rebase conflict

re-enable specs

user vars instead of constants

move non core code in bootstrap

document version files

move version file

remove useless code

use version in logstash-core

fix gem files list

put back concurrent-ruby version constrain as in master

add dependency on logstash-core-event

remove dependency on logstash-core to avoid circular dependency

fix rebase conflict

remove circular dependency

fix specs

update README
2015-11-16 16:40:19 -05:00
Tal Levy
e28f188e12 refactor to gradle project, relates to #4191
more event java updates

- updated metadata
- moved #append to java
- other small test fixes

commit gradle wrapper jar
2015-11-16 16:39:47 -05:00
Suyog Rao
146b3f6c5d Add jdbc input to default package list
JDBC input is a popular plugin and has been stable over the last
few releases.

Fixes #4198
2015-11-16 18:02:54 +00:00
Pier-Hugues Pellerin
a30b9a905e Add the possibility to run local integration test
Fixes #4166
2015-11-13 17:16:49 +00:00
Gabriel Moskovicz
abd97aa98b Add powershell script for windows testing
- Release acceptance test, which tests ports and run Logstash.
- Default plugins, which runs the same plugin installation and
environment for windows
- Simple integration, which integrates Logstash with Elasticsearch and a
simple configuration

Fixes #3994
2015-11-06 10:29:57 -08:00
Pier-Hugues Pellerin
ed97013196 Logstash-input-beats is now part of the default plugins
Fixes #4057
2015-10-19 18:22:22 +00:00
Suyog Rao
bfbb49dc14 Remove elasticsearch_http from default plugins list
Fixes #4002
2015-10-08 13:35:30 +00:00
guyboertje
7bb94d0692 add test for compact and compact! in java_integrations
Fixes #3772
2015-09-03 19:43:10 +00:00
Suyog Rao
568231816e Update ot JRuby 1.7.22
Fixes #3835
2015-09-02 20:54:50 +00:00
Suyog Rao
e2a5e92faf Jenkins: Ignore DynamoDB for now. Make sure all plugins tests pass
Fixes #3839
2015-09-02 17:53:26 +00:00
Pere Urbon-Bayes
028d76497c add the missing magic header # encoding: utf-8 so all internal strings are UTF-8 in Ruby < 2.0
Fixes #3723
2015-08-17 13:30:07 +00:00
Suyog Rao
229c42a964 Bundle http input plugin with defaults
Fixes #3507
2015-06-29 21:19:13 +00:00
Aaron Mildenstein
3d3cd8ab8f Do not include rackspace input/output in tests
Block testing by regex.

fixes #3518

Fixes #3519
2015-06-29 18:03:34 +00:00
Pier-Hugues Pellerin
fd4824c23f Add a micro benchmark runner
Fixes #3457
Add `rake benchmark:run` task to run micro benchmark on specific part of
the code, it's uses the benchmark-ips gem to generate gem.
See https://github.com/evanphx/benchmark-ips for usage
2015-06-26 11:11:52 -07:00
Pere Urbon-Bayes
cf10890d63 fix the coverage analysis throw simplecov to take care of all files
make the eager loading patterns consistent in the spec_helper.rb file

make the setup-simplecov task not a dependency, but an explicit task only executed when ENV['COVERAGE'] is defined

refactor eager loading code plus add some documentation to the setup-simplecov task

Added more comments to the test:setup task

Fixes #3465
2015-06-25 15:36:57 +00:00
Suyog Rao
fb5c388693 Add comma after new line was added
Fixes #3472
2015-06-19 17:09:25 +00:00
Suyog Rao
f26ab64ec4 Add NOTICE.TXT to comply with License
Fixes #3458
2015-06-19 16:47:10 +00:00
Pier-Hugues Pellerin
d85fdcb86d remove benchmark from this PR
Fixes #3425
2015-06-18 17:59:49 +00:00
Pier-Hugues Pellerin
fcf8d783fa Add a caching mechanism for the #sprintf calls
This PR allow the Event class to compile a `sprintf` the first time and
reuse the template for the other calls. This strategy speeds up considerably
calls that uses date formatting or use fieldref its a bit slower if you
dont do any string manipulation.

Fixes #3425
2015-06-18 17:59:47 +00:00
Pere Urbon-Bayes
1849ff23bb add logstash-output-webhdfs as is released, but still not completely integrated with logstash to be run in the all plugins test
Fixes #3401
2015-06-09 09:31:18 +00:00
Pere Urbon-Bayes
06c5a893b5 add logstash-input-http as requested by jsvd
Fixes #3399
2015-06-09 09:05:57 +00:00
Pere Urbon-Bayes
8012c98895 add the perfmon plugin to the skip list
Fixes #3399
2015-06-09 09:05:57 +00:00
Pere Urbon-Bayes
d5fc55b3a6 update is_released to use Gems.info
Fixes #3391
2015-06-08 13:11:26 +00:00
Pere Urbon-Bayes
11d2f7b3ed fix the case where a plugin is released and has a similar name to one still not released, fixing failured on the all plugins test
Fixes #3391
2015-06-08 13:11:26 +00:00
Pere Urbon-Bayes
e2450f1f78 Make the necessary changes to use rspec3 as a default test runner for
logstash, including adapting the test suites that were still using 2.x
syntax.
2015-06-08 12:19:11 +02:00
Colin Surprenant
827fc6d418 update to jruby-1.7.20 and cleanup legacy patches 2015-05-25 16:51:32 -04:00
Pere Urbon-Bayes
012a872745 update rubycheck to use DEBUG insted of as it will never get used because of the relaunch
Fixes #3251
2015-05-15 13:09:22 +00:00
Pere Urbon-Bayes
a84387da68 make the extracting lines for JRUBY optional throw using the DEBUG env variable
Fixes #3251
2015-05-15 13:09:22 +00:00
Jordan Sissel
4b325bd2de Set package epoch to 1
This is done because we made a mistake in the versioning of the rpm and
debs.

rpm and deb both sort "1.5.0.rc4" to be newer than "1.5.0". The default
epoch is 0, so setting epoch to 1 will cause "1.5.0" to sort higher than
the release candidates and betas.
2015-05-14 11:28:44 -07:00
Pere Urbon-Bayes
7675a19032 Few minor improvements and pending cleanups:
* Clean up pending development dependencies from the logstash-core gemspec
* Add gems as a build dependency in the Gemfile, is used when doing an
  install-all task, and failed in the ci env (like octokit)
2015-05-11 16:01:04 +02:00
Pere Urbon-Bayes
6241b7f0ca make the documentation generation task more clean to understand with smaller lines
Fixes #3194
2015-05-08 07:57:09 +00:00
Pere Urbon-Bayes
286dac435e make rubyzip a dependency installed via the Gemfile, so the artifact:zip task works as expected 2015-05-06 15:46:10 +02:00
Colin Surprenant
7a1ec64c23 use :build dependencies
fix octokit dep

fixes #3171
2015-05-06 08:27:57 -04:00
Colin Surprenant
07b6f4533c support remote or local gem packaging
typo
2015-05-04 20:50:44 -04:00
Colin Surprenant
48bcc5b74d cleanup dependencies and rely on logstash-core gem
remove unused load path

use either local core lib dir or logstash-core gem

include spec in logstash-core

do not include lib, spec and locales dirs in package root

environment bootstrapping in lib/bootstrap.rb

added comments

move pluginmanager out of logstash-core

kafla installation solved by pluginmanager refactor

refactor bootstrap code

refactor shell scripts to simplify and DRY, indirect rspec command to our ruby launcher

add bin/rspec

cut by half the bin/plugin and bin/rspec startup time

fix drip context

use printf instead of tr

updated Windows bin scripts

missing gemspec in gemspec

use gem instead of gemspec so our plugin manager can correctly install logstash-core

generate packages including locally built logstash-core gem

move jenkins developmnent dependencies into Gemfile, they do not belong in logstash-core

path leftover

clean help agent help usage message and remore rspec in available command

comments cosmetics

update Bundler dependency, all recent testing have been made with 1.9.4

updated .lock file with regard to updated Gemfile

cleanup gemspec, Gemfile and regen .lock file

added progress output

avoid dual require on version

closes #3096
2015-05-04 18:17:22 -04:00
Pere Urbon-Bayes
a01ffd4ee6 slim down the skip list from plugins that are now released and accesible
Fixes #3128
2015-04-29 06:25:25 +00:00
Colin Surprenant
3dabba80fa revert multi_filter removal and add check for cancelled events
add filter and flush compiles functions specs

missing multiline filter for core specs

fixes #3100
2015-04-27 13:54:18 -04:00
Colin Surprenant
c70f252fe4 remove unwanted files in package 2015-04-22 18:49:35 +02:00
Colin Surprenant
4e1246a5b5 remove freeze task 2015-04-22 18:49:21 +02:00
Colin Surprenant
b4c3d1ef33 upgrade jruby to 1.7.19
Fixes #3066
2015-04-21 15:44:35 +00:00
Pere Urbon-Bayes
0b72d06de8 add an rspec shim to complement the old bin/logstash rspec way of running specs in logstash
simplify the rspec shim codebase

add bin/rspec to the exclude list for packaging

removed leftover

Fixes #3047
2015-04-20 15:47:46 +00:00
Colin Surprenant
bde2691239 bin/bundle shim
use  instead for $:

exclude bin/bundle from packages

added comment
2015-04-20 17:07:52 +02:00