* plugin manager: add --no-expand flag for list command
Allows us to avoid expanding aliases and integration plugins
* spec: escape expected output in regexp
Removal works in a single pass by finding plugins that would have unmet
dependencies if all of the specified plugins were to be removed, and
proceeding with the removal only if no conflicts were created.
> ~~~
> ╭─{ rye@perhaps:~/src/elastic/logstash@main (pluginmanager-remove-multiple ✘) }
> ╰─● bin/logstash-plugin remove logstash-input-syslog logstash-filter-grok
> Using system java: /Users/rye/.jenv/shims/java
> Resolving dependencies......
> Successfully removed logstash-input-syslog
> Successfully removed logstash-filter-grok
> [success (00:00:05)]
~~~
handle maven output that can carry "garbage" information after the jar's name. this patch deletes that extra information, also pins psych to 5.2.2 until jruby ships with snakeyaml-engine 2.9 and jar-dependencies 0.5.2
Related to: https://github.com/jruby/jruby/issues/8579
The preferred channel for communication about LS is the elastic discussion
forum, this commit updates the source code and readme files to reflect that.
* Exclude plugins feature in OSS distributions.
* Set elastic_integration plugin default.
* Remove non-OSS plugins after installing default plugins.
* Testing local can't find gem bundler (= 2.3.26) issue.
* Include extract non-OSS plugins logic indocker build operations.
* Only default plugins can be excluded from OSS distros.
* Simplification: instead conditional check, use intersection to make OSS exlucluded plugin list.
* Gem and specification files still stay after removing the plugin. This change removes the stayed files.
* Rename oss-exclude to skip-oss to align namings with other params.
* Make intersection method simpler.
* [Test] Temporary excluding elastic integration plugin from default plugin list.
* Sets elastic_integration plugin default back. When removing locally installed gems, Gem::Specification doesn't recognize the gem. We have Bundle::setup in the removal logic but it is causing an issue when we re-use the bundle.
* Test the build order, remove plugin from cache logic seems invalid since we don't pack the cache.
This commit added support to add and remove multiple keystore keys in a single operation. It also fixed the empty value validation for editing existing key values and added ASCII validation for values.
* deps: downgrade jruby, keep updated default-gem dependencies (#15283)
forward-ports non-release-branch components of #15283 to `main`
* deps: downgrade jruby, keep updated default-gem dependencies
By downgrading JRuby to 9.4.2.0 we avoid the silent global crash of the
scheduler backing `Concurrent::TimerTask` that occurs when Jruby 9.4.3.0's
invokedynamic promotes a method to run natively, incorrectly.
Upstream bug: https://github.com/jruby/jruby/issues/7904
Along with the downgrade of JRuby itself to 9.4.2.0, we cherry-pick the
updates to gems that were included in the latest JRuby 9.4.3.0 to ensure
we don't back out relevant fixes to stdlib.
We also remove a pinned-dependency on `racc` that is no longer relevant.
Resolves: https://github.com/elastic/logstash/issues/15282
* Imported the licenses for some gems
- cgi
- date
- ffi-binary-libfixposix
- io-console
- net-http
- net-protocol
- reline
- time
- timeout
- uri
* specs: avoid mocking global ::Gem::Dependency::new
* build: remove redundanct dependsOn declaration
* deps: notice use of ffi-binary-libfixposix via Ruby license
this gem is tri-licensed `Ruby` / `EPL-2.0` / `LGPL-2.1-or-later` and
the Ruby license is preferred to EPL when available
---------
Co-authored-by: andsel <selva.andre@gmail.com>
* deps: add license notices for gems moved from default to bundled
---------
Co-authored-by: andsel <selva.andre@gmail.com>
When using a proxy with the plugin manager, this template is written to `~/.m2/settings.xml`. The
license header is also copied, which generates invalid XML which maven cannot parse.
Set of changes to make Logstash compatible to JRuby 9.4.
Bundle JRuby 9.4.3.0
- Redefine space token in `LSCL` and `grammar` treetop from `_` which would generated methods in the form `def _0` (deprecated since `2.7`) to `sc`.
- `I18n.t` method doesn't accept hash as second argument
- `URI.encode` has been replaced with same functionality with `URI::Parser.new.escape`
- `YAML.load` needs explicit `fallback: false` to return false when the yaml string is empty (or contains only comments)
- JRuby's `JavaClass` has been removed, now it can use `java.lang.Class` directly
- explicitly require gem `thwait` to satisfy `require "thwait"` (In `Gemfile.template` and `logstash-core/logstash-core.gemspec`)
- fix not args `clone` to be `def clone(*args)`
- fix `Enumeration.each_slice` which from `Ruby 3.1` is [chainable](https://rubyreferences.github.io/rubychanges/3.1.html#enumerableeach_cons-and-each_slice-return-a-receiver) and doesn't return `nil`. JRuby fixed in https://github.com/jruby/jruby/issues/7015
- Expanded `Down.download` arguments map ca16bbed3c302006967413eb9d3862f2da81f7ae
- Avoid to pass `nil` in the list of couples used in `Hash[ <list of couples> ]` which from Ruby `3.0` generates an `ArgumentError`
- Removed space not allowed between method name and parentheses `initialize (` is forbidden. 29b607dcdef98f81a73ad171639fd13aaa65e243
- With [Ruby 2.7 the `Kernel#open`](https://rubyreferences.github.io/rubychanges/2.7.html#network-and-web) doesn't fallback to `URI#open`, fixed test code that used that to verify open port. e5b70de54c5301f51a767da67294092af0cfafdc
- Avoid to drop `rdoc/` folder from vendored JRuby else `bin/logstash -i irb` would crash, commit b71f73e9c6edb81a7b7ae1305047e506f61c6e8c
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Revert "Revert "Unpin bundler to allow 2.4 (#14894)" (#14942)"
This reverts commit 5e3038a3d3.
* prevent bundler 2.4 from blowing up memory during ./gradlew generatePluginsVersion
for some reason, instances of Bundler::Dsl are retained in
org.jruby.MetaClass. Each Bundler::Dsl will retain Bundler::SourceList
which is > 150MB. This commit sets the internal state of each Dsl's
source list so the large objects can be collected.
The proper fix would be to either reuse a single Dsl object or fix the
code upstream to avoid having Dsl objects retained by org.jruby.MetaClass.
* skip input cloudwatch during generatePluginsVersion
the new bundler 2.4's resolution algorithm seems to struggle in certain
situations, like resolving combination of "logstash-input-cloudwatch" and
"logstash-integration-aws".
Until the issue is solved let's skip it.
This reverts commit 6d08a7c1cd.
Bundler 2.4 is accumulating memory during the `generateDocsVersion` task
outside of the known Bundler 2.3 DepProxy class.
Until the memory mitigation patch is adapted to Bundler 2.4 it's best to revert 6d08a7c1cd.
* Make generated input plugin immediately buildable
* Apply the same changes to codec, filter, and output templates
* Uncomment TODOs in example gemspecs
When run in debug mode, #invoke was returning an instance of UI::Shell rather
than a string, causing the plugin to crash when `<<` was called on.
This commit ensures that a string is returned regardless of whether debug is set
Fixes: #14131
This commit updates the version of jruby used in Logstash to `9.3.4.0`.
* Updates the references of `jruby` from `9.2.20.1` to `9.3.4.0`
* Updates references/locations of ruby from `2.5.0` to `2.6.0`
* Updates java imports including `org.logstash.util` to be quoted
* Without quoting the name of the import, the following error is observed in tests:
* `java.lang.NoClassDefFoundError: org/logstash/Util (wrong name: org/logstash/util)`
* Maybe an instance of https://github.com/jruby/jruby/issues/4861
* Adds a monkey patch to `require` to resolve compatibility issue between latest `jruby` and `polyglot` gem
* The addition of https://github.com/jruby/jruby/pull/7145 to disallow circular
causes, will throw when `polyglot` is thrown into the mix, and stop logstash from
starting and building - any gems that use an exception to determine whether or not
to load the native gem, will trigger the code added in that commit.
* This commit adds a monkey patch of `require` to rollback the circular cause exception
back to the original cause.
* Removes the use of the deprecated `JavaClass`
* Adds additional `require time` in `generate_build_metadata`
* Rewrites a test helper to avoid potentially calling `~>` on `FalseClass`
Co-authored-by: Joao Duarte <jsvduarte@gmail.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* A structural change for aliased plugin declarations to support doc header replaces.
* YAML file structure made to be more readable, make AliasPlugin class immutable.
* Do not expose alias doc header replaces since we are not using them. Unit tests failure fix.
* Refactor: logstash boot to know what to do
* Refactor: reduce LOAD_PATH - load jars directly
* Refactor: unused requires in runner
* Refactor: boot - require libs when to be used
After the fix of unlocking ecs_compatibility_support version in plugin update (#13218), `logstash-plugin install` has a problem of installing non default plugin.
This commit removes `Bundler.setup` in install path to avoid Gemfile froze by bundler
Fixed: #13404
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Fix bundler handling of 'without'
Prior to this change, the values set in `set_local` are ignored when invoking
bundler via the command line, as is used with `invoke!`. This commit sets those
values in `ENV` variables instead, fixing the functionality to not install
development gems.
* Update bundler spec to check ENV variable
* Added test to ensure kramdown gem not vendored
* Re-add set_local setting to play nice with `expand_logstash_mixin_dependencies`
* logstash service needs to be installed
* gem_vendored? needs to use full path to vendor files
* use `stdout` from `cat` command to generate spec temporary file
* Removed unnecessary support for supplying a block from #gem_vendored?
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
This commit fixes the `logstash-plugin update` command which fail to update plugin
that depends on a new version of logstash-mixin-ecs_compatibility_support.
It resolves logstash-* dependencies and puts them in bundler update command.
Fixed: #13181
This PR enables the upgrade of bundler to the latest version.
Prior to this PR, the ability to do so was blocked by bundler.setup in versions of bundler > `2.23` making runtime changes to `Gemfile.lock` (unless the lock file was `frozen`) based on the specific platform the application was being run on, overriding any platforms (including generic `java` platform) set during build time. This was in conflict with changes made in #12782, which prevented the logstash user writing to files in `/usr/share/logstash`.
This PR will freeze the lockfile when logstash is run, and unfreeze it when manipulating plugins (install, update, remove, install from offline pack) to allow new plugins to be added. While unfrozen, changes are also made to ensure that the platform list remains as the generic `java` platform, and not changed to the specific platform for the runtime JVM.
This PR also introduces a new runtime flag, `--enable-local-plugin-development`. This flag is intended for use by Logstash developers only, and enables a mode of operation where a Gemfile can be manipulated, eg
```
gem "logstash-integration-kafka", :path => '/users/developer/code/plugins/logstash-integration-kafka'
```
to facilitate quick and simple plugin testing.
This PR also sets the `silence_root_warning` flag to avoid bundler printing out alarming looking warning messages when `sudo` is used. This warning message was concerning for users - it would be printed out during normal operation of `bin/logstash-plugin install/update/remove` when run under `sudo`, which is the expected mode of operation when logstash is installed to run as a service via rpm/deb packages.
This PR also updates the vagrant based integration tests to ensure that Logstash still runs after plugin update/install/remove operations, fixes up some regular expressions that would cause test failures, and removes some dead code from tests.
## Release notes
* Updated Bundler to latest version
* Ensured that `Gemfile.lock` are appropriately frozen
* Added new developer-only flag to facilitate local plugin development to allow unfrozen lockfile in a development environment
Remove hard coded alias definitions in favor of yaml descriptor file.
Introduce a single point of aliases definition (logstash-core/src/main/resources/org/logstash/plugins/AliasRegistry.yml), checksum and copy it around to be used by Logstash and by Logstash's plugin management tool.
The descriptor yml file contains a checksum to verify it's not changed accidentally in a deployment of Logstash, if the verification phase fail Logstash avoid to start and plugin management tool avoid to operate.
The signing and copying around is managed by a specific Gradle task invoked during the build.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Fixes#12831
Adapted install/uninstall/list PluginManager's CLI commands to respect the alised plugins
- adapt install plugin to resolve an alias giving precedence on a real plugin
- changed list to mark alised plugins
- uninstall avoid to remove the alias and ask the user to remove the original plugin
- update update the original plugin in case of alias, else fallback on usual behavior
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
