Adds log.format.json.fix_duplicate_message_fields feature flag to rename the clashing fields when json logging format (log.format) is selected.
In case two message fields clashes on structured log message, then the second is renamed attaching _1 suffix to the field name.
By default the feature is disabled and requires user to explicitly enable the behaviour.
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
Introduce a new setting named `pipeline.buffer.type` which could be valued direct or heap to enable the allocation on Java heap.
The processing of the setting is done in `LogStash::Runner#execute` and sets the Java properties considered by Netty to disable the direct allocation: `io.netty.noPreferDirect`.
However, if that system property is already configured explicitly by the user (because set in `jvm.options`or `LS_JAVA_OPTS`) the setting doesn't take place and warning log is reported, respecting the user's will.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Describe how to troubleshoot connectivity problems at HTTP level with Azure Blob Storage
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
This commit added support to add and remove multiple keystore keys in a single operation. It also fixed the empty value validation for editing existing key values and added ASCII validation for values.
* Update the Logstash to Logstash Native doc to reflect the multiple hosts usage.
* Logstash to Logstash comm page, adding LS-to-LS native HA support.
* Apply suggestions from code review
Refining the context.
Co-authored-by: Andres Rodriguez <andreserl@gmail.com>
---------
Co-authored-by: Andres Rodriguez <andreserl@gmail.com>
* geoip: extract database manager to stand-alone feature
Introduces an Elastic-licensed GeoipDatabaseManagement tool that can be used
by ANY plugin running on Elastic-licensed Logstash to retrieve a subscription
to a GeoIP database that ensures EULA-compliance and frequent updates, and
migrates the previous Elastic-licensed code-in-Logstash-core extension to
the Geoip Filter to use this new tool, requiring ZERO changes to in-the-wild
versions of the plugin.
The implementation of the new tool follows the previous implementation as
closely as possible, but presents a new interface that ensures that a
consumer can ATOMICALLY subscribe to a database path without risk that the
subscriber will receive an update or expiry before it is finished applying
the initial value:
~~~ ruby
geoip_manager = LogStash::GeoipDatabaseManagement::Manager.instance
subscription = geoip_manager.subscribe('City')
subscription.observe(construct: ->(initial_dbinfo){ },
on_update: ->(updated_dbinfo){ },
on_expire: ->( _ ){ })
subscription.release!
~~~
* docs: link in geoip database manager docs
* docs: reorganize pending 'geoip database management' feature
* docs: link to geoip pages from feature index
* geoip: add SubscriptionObserver "interface"
simplifies using Subscription#observe from Java
* geoip: fixup SubscriptionObserver after rename
* geoip: quacking like a SubscriptionObserver is enough
* geoip: simplify constants of legacy geoip filter extension
* geoip: bump logging level to debug for non-actionable log
* geoip: refine log message to omit non-actionable info
* re-enable invokedynamic (was disabled to avoid upstream bug)
* geoip: resolve testing fall-out from filter extension's "private" constants removal
* geoip: consistently use `DataPath#resolve` internally, too
* Add known imap and email plugin issues section to Logstash 8.10+ versions.
Co-authored-by: Mashhur <mashhur.sattorov@elastic.co>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
(cherry picked from commit a88f82e77f)
Expands the description of memory used by Logstash, dividing the heap and non-heap; describing in details which parts composes the non-heap, how to size it and list the JVM settings that can be used to properly define this space.
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Since the DRA build for 8.10.1 was made with 82daae80bb , this fix didn't get in.
(cherry picked from commit aa9265665e)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
