The difference is that our matcher uses .find() to search for a regex match anywhere in the string, whereas the hamcrest one uses .matches() to check the whole string against the regex. This leads to more specific regex checks.
I've left our own one for YAML tests, as that way we don't need to mangle the regex to add .* either side, which might be confusing in test failures.
in 2.17.2 (patch release) log4j has made a refactoring that requires a Configuration to be manually
passed into the created PatternLayout
If the Configuration is not passed, the System Variable lookup will not work
This results in cluster.name field not being populated in logs if ESv7 layout was used on ESv8 cluster (cloud)
This commit creates a PatternLayout with a DefaultConfiguration (the same was used previous to the refactoring)
Also adds back tests that verifies ESv8 cluster with ESv7 logging config. We still have that situation in the cloud so it is better to have tests around this.
relates apache/logging-log4j2#1551
Use local-independent `Strings.format` method instead of `String.format(Locale.ROOT, ...)`.
Inline `ESTestCase.forbidden` calls with `Strings.format` for the consistency sake.
Add `Strings.format` alias in `common.Strings`
The dataset value for all ES logs are prefixed with elasticsearch + log
type. Like elasticsearch.server. Deprecation log had it reverted
deprecation.elasticsearch
This commit renames the dataset for deprecation logs to
`elasticsearch.deprecation.
closes#83251
When bootstrapping Elasticsearch, the console appender is automatically
added by our log4j config. However, there are some cases where we want
to remove that appender. First, if --quiet is passed, we do not want to
log anything to it. Second, if we are daemonizing, then we want to close
the streams, so we need to remove it if it exists. Third, when huge
guice/startup exceptions occur, we log these only to the ES log file, so
we need to remove the appender temporarily.
This commit moves the logic for mucking with the console appender into
LogConfigurator. In the future this can be better isolated within
logging, perhaps even avoiding creating the console appender to begin
with when using --quiet, but for now this at least gets some log specific
logic out of bootstrap.
relates #85758
The ES code base is quite JSON heavy. It uses a lot of multi-line JSON requests in tests which need to be escaped and concatenated which in turn makes them hard to read. Let's try to leverage Java 15 text blocks for representing them.
This commit emits the value of x-elastic-product-origin header into elasticsearch.elastic_product_origin field in ES deprecation logs and indexed deprecation logs.
x-elastic-product-origin header is intended to identify the elastic stack origin and allow to ignore deprecations emitted by the stack.
This commit updates all deprecation message (except for REST
compatible API messages) in 8.0+ to be emit at warning level.
Currently none of these have been removed in future versions (yet) so they
should be logged at warning, not critical.
This commit also changes the default assertWarning to assert at warning level
and introduces a new assertCriticalWarning to assert critical warnings.
A recent change for the deprecation logs provided the capability to emit deprecation's at critical vs. warning levels, #77482.
However deprecated settings always log at critical level without the ability to express that the setting deprecation is only a
warning.
This commit exposes the ability to set the deprecation level when deprecating a setting.
Closes#78781
Fix the split package org.elasticsearch.common.xcontent, between server and the x-content lib. Move the x-content lib exported package from org.elasticsearch.common.xcontent to org.elasticsearch.xcontent ( following the naming convention of similar libraries ). Removing split packages is a prerequisite to modularization.
We do not need to assert about custom configurations or previous version
configurations. Users should update their config to latest version when
upgrading.
closes#77430
This commit changes default deprecation logger level to CRITICAL, where default means deprecations emitted by DeprecationLogger#critical method.
It also introduces WARN deprecations which are emitted by DeprecationLogger#warn Those log lines emitted at WARN are meant to indicate that a functionality is deprecated but will not break at next major version.
relates #76754
This adds just enough support for the traceparent header to be useful in es8.
Since Elasticsearch already logs in ECS format extending it with support for transaction.id and trace.id is a quick win.
This allows us to surface server/deprecation slow logs from an instrumented application using the Trace Logs feature.
Parsing `traceparent` in http layer and populating tasks with `trace_id` which is preserved in thread context.
ParseField is part of the x-content lib, yet it doesn't exist under the
same root package as the rest of the lib. This commit moves the class to
the appropriate package.
relates #73784
When libs/core was created, several classes were moved from server's
o.e.common package, but they were not moved to a new package. Split
packages need to go away long term, so that Elasticsearch can even think
about modularization. This commit moves all the classes under o.e.common
in core to o.e.core.
relates #73784
Extract usage of internal API from TestClustersPlugin and PluginBuildPlugin and related plugins and build logic
This includes a refactoring of ElasticsearchDistribution to handle types
better in a way we can differentiate between supported Elasticsearch
Distribution types supported in TestCkustersPlugin and types only supported
in internal plugins.
It also introduces a set of internal versions of public plugins.
As part of this we also generate the plugin descriptors now.
As a follow up on this we can actually move these public used classes into
an extra project (declared as included build)
We keep LoggedExec and VersionProperties effectively public And workaround for RestTestBase
* Warn users if security is implicitly disabled
Elasticsearch has security features implicitly disabled by default for
Basic and Trial licenses, unless explicitly set in the configuration
file.
This may be good for onboarding, but it also lead to unintended insecure
clusters.
This change introduces clear warnings when security features are
implicitly disabled.
- a warning header in each REST response if security is implicitly
disabled;
- a log message during cluster boot.
A #68808 introduced a possibility to declare fields which will be only available to parsing when a compatible API was used.
This commit replaces deprecated log with compatible logging when a 'compatible only' field was used. Also includes a refactoring of LoggingDeprecationHandler method names
relates #51816
The commit #55115 removed the possibility to directly force deprecation
log to be emitted. This means that usage of deprecated fields was
throttled and only one deprecation was logged. The key was common for
all fields = "deprecated_field".
This commit appends a used deprecated field name to prevent that
throttled.
Extending deprecation logger to also log compatible API warnings.
These warnings will be written to deprecation log file, data stream and will emit response warning header - the same as deprecation logs. In order to comply with constant keyword type of data_stream.datatype the value will have to be the same for compatible and deprecation log - elasticsearch.deprecation.
The compatible and deprecation logs will differ by elasticsearch.event.category field which will always be DeprecationCategory.COMPATIBLE_API for all compatible logs
closes#52369
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:
- Updating LICENSE and NOTICE files throughout the code base, as well
as those packaged in our published artifacts
- Update IDE integration to now use the new license header on newly
created source files
- Remove references to the "OSS" distribution from our documentation
- Update build time verification checks to no longer allow Apache 2.0
license header in Elasticsearch source code
- Replace all existing Apache 2.0 license headers for non-xpack code
with updated header (vendored code with Apache 2.0 headers obviously
remains the same).
- Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
Closes#64824. Introduce the concept of categories to deprecation
logging. Every location where we log a deprecation message must now
include a deprecation category.
This ports the majority of the rest integ tests tasks to use the task avoidance api.
- There are some edge cases left that we need to investigate, but we can do that separately.
datastream related fields should be always be provided.
message field is already set with a lazy value (Object.toString) within
ESLogMessage
Also removing test cases to directly access deprecation logger with LogManger.getLogger("deprecation"). This is disallowed. DeprecationLogger.getLogger should be used.
Part of #46106. Simplify the implementation of deprecation logging by
relying of log4j more completely, and implementing additional behaviour
through custom appenders and filters.
* Merge test runner task into RestIntegTest
* Reorganizing Standalone runner and RestIntegTest task
* Rework general test task configuration and extension
DeprecationLogger's constructor should not create two loggers. It was
taking parent logger instance, changing its name with a .deprecation
prefix and creating a new logger.
Most of the time parent logger was not needed. It was causing Log4j to
unnecessarily cache the unused parent logger instance.
Splitting DeprecationLogger into two. HeaderWarningLogger - responsible for adding a response warning headers and ThrottlingLogger - responsible for limiting the duplicated log entries for the same key (previously deprecateAndMaybeLog).
Introducing A ThrottlingAndHeaderWarningLogger which is a base for other common logging usages where both response warning header and logging throttling was needed.
relates #55699
relates #52369
Updating log4j2 configuration to use `EcsLayout` for our JSON logs https://github.com/elastic/ecs-logging-java instead of previously used `ESJsonLayout`
ESJsonLayout is still supported, but will be marked as deprecated.
Deprecation, Indexing and Search slow logs are in ECS JSON format only. Plaintext is no longer backwards compatible and will result in different output.
Server logs are in both plaintext and ECS JSON format.
relates #49087
This commit changes the ThreadContext to just use a regular ThreadLocal
over the lucene CloseableThreadLocal. The CloseableThreadLocal solves
issues with ThreadLocals that are no longer needed during runtime but
in the case of the ThreadContext, we need it for the runtime of the
node and it is typically not closed until the node closes, so we miss
out on the benefits that this class provides.
Additionally by removing the close logic, we simplify code in other
places that deal with exceptions and tracking to see if it happens when
the node is closing.
Closes#42577
Closes#48724. Update `.editorconfig` to make the Java settings the default
for all files, and then apply a 2-space indent to all `*.gradle` files.
Then reformat all the files.
* refactor ESLogMessage and overriding fields
* rename esmessagefields to overrideFields
* javadoc
* simplify custom message class creation
* line length
* small cleanup and rename of ESJsonField
* working with builder
* parametrized map - basing on ecs
* builder
* refactoring and cleanup
* cleanup
* logger checker
* Mute failing test
tracked in #44552
* mute EvilSecurityTests
tracking in #44558
* Fix line endings in ESJsonLayoutTests
* Mute failing ForecastIT test on windows
Tracking in #44609
* mute AutoFollowIT.testConflictingPatterns
tracking in #44610
* mute BasicRenormalizationIT.testDefaultRenormalization
tracked in #44613
* Revert "mute AutoFollowIT.testConflictingPatterns"
This reverts commit 012de08f59.
* mute x-pack internal cluster test windows
tracking #44610
* Mute failure unconfigured node name
* fix mute testDefaultRenormalization
* Increase busyWait timeout windows is slow
* Mute JvmErgonomicsTests on windows
Tracking #44669
* mute SharedClusterSnapshotRestoreIT testParallelRestoreOperationsFromSingleSnapshot
Tracking #44671
* Mute NodeTests on Windows
Tracking #44256
Deprecation logger was filtering log entries by key, that means that if two log messages with the same key are logged from different users, then the second log messages will be filtered.
This change allows to log deprecation message with the same key by different users.
relates #41354
This is a refactor to current JSON logging to make it more open for extensions
and support for custom ES log messages used inDeprecationLogger IndexingSlowLog , SearchSLowLog
We want to include x-opaque-id in deprecation logs. The easiest way to have this as an additional JSON field instead of part of the message is to create a custom DeprecatedMessage (extends ESLogMEssage)
These messages are regular log4j messages with a text, but also carry a map of fields which can then populate the log pattern. The logic for this lives in ESJsonLayout and ESMessageFieldConverter.
Similar approach can be used to refactor IndexingSlowLog and SearchSlowLog JSON logs to contain fields previously only present as escaped JSON string in a message field.
closes#41350
We had this as a dependency for legacy dependencies that still needed
the Log4j 1.2 API. This appears to no longer be necessary, so this
commit removes this artifact as a dependency.
To remove this dependency, we had to fix a few places where we were
accidentally relying on Log4j 1.2 instead of Log4j 2 (easy to do, since
both APIs were on the compile-time classpath).
Finally, we can remove our custom Netty logger factory. This was needed
when we were on Log4j 1.2 and handled logging in our own unique
way. When we migrated to Log4j 2 we could have dropped this
dependency. However, even then Netty would still pick up Log4j 1.2 since
it was on the classpath, thus the advantage to removing this as a
dependency now.
In hamcrest 2.1 warnings for unchecked varargs were fixed by hamcrest using @SafeVarargs for those matchers where this warning occurred.
This PR is aimed to remove these annotations when Matchers.contains ,Matchers.containsInAnyOrder or Matchers.hasItems was used
This commit replaces the existing RandomizedTestingTask and supporting code with Gradle's built-in JUnit support via the Test task type. Additionally, the previous workaround to disable all tasks named "test" and create new unit testing tasks named "unitTest" has been removed such that the "test" task now runs unit tests as per the normal Gradle Java plugin conventions.
This commit replaces the existing RandomizedTestingTask and supporting code with Gradle's built-in JUnit support via the Test task type. Additionally, the previous workaround to disable all tasks named "test" and create new unit testing tasks named "unitTest" has been removed such that the "test" task now runs unit tests as per the normal Gradle Java plugin conventions
