This change uses a new utility class that
abstracts the construction of secure settings,
into environment dependent version. KeyStoreWrapper
is just one implementation.
When we upgraded to lucene 9.5 (snapshot) with #92957 we initially disable panama-based
mmap directory through a system property. With this commit we remove the system property
and enable java 19 memory segments by default (based on https://github.com/apache/lucene/pull/12033)
Previously UserExceptions could be thrown from the main Elasticsearch
process during startup. With recent refactorings all remaining
validation of arguments was moved to the server cli. Since there are no
longer any uses of the user exception in server, the marker code is no
longer needed. This commit removes that marker and the associated code
in the server cli error pump.
stable plugins not build with ES's gradle plugin will not have named_components.json file.
To allow these plugins to expose their named components, a scan can be performed upon install.
relates #88980
With this commit we specify `level` in JVM logs. `level` helps to filter
more severe messages from mere debugging messages in logs and with this
change we are able to use it also for JVM logs.
Here are a few example lines:
Before:
```
[2022-12-15T06:19:16.936+0000][15181][gc,start ] GC(0) Pause Young (Normal) (G1 Evacuation Pause)
```
After:
```
[2022-12-15T06:22:27.932+0000][16320][info][gc,start ] GC(0) Pause Young (Normal) (G1 Evacuation Pause)
[2022-12-15T06:22:27.932+0000][16320][debug][gc,age ] GC(0) Desired survivor size 14680064 bytes, new threshold 15 (max threshold 15)
[2022-12-15T06:22:27.935+0000][16320][info ][gc,phases ] GC(0) Pre Evacuate Collection Set: 0.1ms
[2022-12-15T06:22:27.935+0000][16320][trace][gc,age ] GC(0) Age table with threshold 15 (max threshold 15)
```
Notice, that the log level might have trailing spaces.
See also
https://docs.oracle.com/en/java/javase/19/docs/specs/man/java.html#xlog-tags-and-levels
for the available log levels.
Use local-independent `Strings.format` method instead of `String.format(Locale.ROOT, ...)`.
Inline `ESTestCase.forbidden` calls with `Strings.format` for the consistency sake.
Add `Strings.format` alias in `common.Strings`
This commit adds a new test framework for configuring and orchestrating
test clusters for both Java and YAML REST testing. This will eventually
replace the existing "test-clusters" Gradle plugin and the build-time
cluster orchestration.
* Add support for building against defined build-ids of dependent artifacts
* Introduce DraResolvePlugin and functional tests
* Declare initial dra snapshot ci job for elasticsearch
We run the DRA builds against ubuntu as other release related ci jobs have in the past. Also we only have libs2xmlutills available on our ubuntu based ci workers
When we launch Elasticsearch with the APM monitoring
agent, we create a temporary configuration file to
securely pass the API key or secret. This temporary
file is cleaned up on Elasticsearch Node creation.
After we renamed the APM module, the delete logic
didn't get updated, which means we never delete the file
anymore.
This commit:
- fixes the APM module pattern match when we delete
- adds additional delete safety net on failed node start
- adds tests for ensuring the naming dependency isn't
broken again.
Removing the custom dependency checksum functionality in favor of Gradle build-in dependency verification support.
- Use sha256 in favor of sha1 as sha1 is not considered safe these days.
Closes https://github.com/elastic/elasticsearch/issues/69736
Part of #84369. Implement the `Tracer` interface by providing a
module that uses OpenTelemetry, along with Elastic's APM
agent for Java.
See the file `TRACING.md` for background on the changes and the
reasoning for some of the implementation decisions.
The configuration mechanism is the most fiddly part of this PR. The
Security Manager permissions required by the APM Java agent make
it prohibitive to start an agent from within Elasticsearch
programmatically, so it must be configured when the ES JVM starts.
That means that the startup CLI needs to assemble the required JVM
options.
To complicate matters further, the APM agent needs a secret token
in order to ship traces to the APM server. We can't use Java system
properties to configure this, since otherwise the secret will be
readable to all code in Elasticsearch. It therefore has to be
configured in a dedicated config file. This in itself is awkward,
since we don't want to leave secrets in config files. Therefore,
we pull the APM secret token from the keystore, write it to a config
file, then delete the config file after ES starts.
There's a further issue with the config file. Any options we set
in the APM agent config file cannot later be reconfigured via system
properties, so we need to make sure that only "static" configuration
goes into the config file.
I generated most of the files under `qa/apm` using an APM test
utility (I can't remember which one now, unfortunately). The goal
is to setup up a complete system so that traces can be captured in
APM server, and the results in Elasticsearch inspected.
The new stable plugin api will have a slightly different descriptor file
format. This commit prepares for stable plugins by adding support for
reading those new files. The basic info for a plugin is the same like
name and version info. Other stuff like classname are not necessary. The
one additional property specific to the new plugins is "modular", which
indicates whether the jars of the plugin should be loaded as named
modules (this is akin to setting the module path when running java).
As part of #50277, we removed the `TAKE_FILE_OWNERSHIP` option from the
Docker entrypoint script and the associated chroot calls, and instead
just defaulted to running the image as `elasticsearch` instead of
`root`.
However, we didn't check that it was still possible to pass CLI options
to Elasticsearch via CLI arguments, and broke this by mistake. This is
probably an uncommon pattern, versus environment variables or a config
file. Nevertheless, it is supposed to be possible and is mentioned in
the documentation.
Fix the problem by suppling the missing positional params when calling
Elasticsearch, and add a test case so that we don't break it again.
As part of #50277, we removed the `TAKE_FILE_OWNERSHIP` option from the
Docker entrypoint script and the associated chroot calls, and instead
just defaulted to running the image as `elasticsearch` instead of
`root`.
However, we didn't check that it was still possible to pass CLI options
to Elasticsearch via CLI arguments, and broke this by mistake. This is
probably an uncommon pattern, versus environment variables or a config
file. Nevertheless, it is supposed to be possible and is mentioned in
the documentation.
Fix the problem by suppling the missing positional params when calling
Elasticsearch, and add a test case so that we don't break it again.
This changes the LoggedExec task to be configuration cache compatible. We changed the implementation
to use `ExecOperations` instead of extending `Exec` task. As double checked with the Gradle team this task
is not planned to be made configuration cache compatible out of the box anytime soon.
This is part of the effort on https://github.com/elastic/elasticsearch/issues/57918
This adds the generation and upload logic of Gradle dependency graphs to snyk
We directly implemented a rest api based snyk plugin as:
the existing snyk gradle plugin delegates to the snyk command line tool the command line tool
uses custom gradle logic by injecting a init file that is
a) using deprecated build logic which we definitely want to avoid
b) uses gradle api we avoid like eager task creation.
Shipping this as a internal gradle plugin gives us the most flexibility as we only want to monitor
production code for now we apply this plugin as part of the elasticsearch.build plugin,
that usage has been for now the de-facto indicator if a project is considered a "production" project
that ends up in our distribution or public maven repositories. This isnt yet ideal and we will revisit
the distinction between production and non production code / projects in a separate effort.
As part of this effort we added the elasticsearch.build plugin to more projects that actually end up
in the distribution. To unblock us on this we for now disabled a few check tasks that started failing by applying elasticsearch.build.
Addresses #87620
The ingest attachment processor is currently available as a plugin. This
commit moves the processor to the default distribution so it is always
available.
Bootstrap plugins were an internal mechanism added to allow a
filesystemprovider for cloud with the quota-aware-fs plugin. Since that
was removed, bootstrap plugins no longer serve a purpose. They were
never officially documented because they were for internal use only.
This commit removes the bootstrap plugins infrastructure.
The keystore format has been changed a few times since it was first
introduced. Part of Elasticsearch startup automatically upgrades the
format. Since Elasticsearch has fixed bounds of supported versions for
upgrades, there are also fixed bounds on the keystore formats we might
need to read.
The v3 keystore format was introduced in Elasticsearch 6.3.0. Since
current Elasticsearch master branch is 8.x, and 8.x only supports
offline upgrades from 7.x, it is therefore impossible to need to read
v1 or v2 formats. This commit removes support for those formats.
During startup of Elasticsearch we go to great lengths to present errors
in a meaningful way to users. Over time, though, the error handling has
been amended to address various issues, and that has resulted in a
complicated system of try/catches handling various cases. One
particularly kludgy piece is removing the console logger in special
cases to avoid printing exceptions to the console. Additionally, the
console removal wasn't actually effective because later in exception
handling the exception would be both logged anyways, and then also sent
to stderr, meaning that we could see the same exception several times.
This commit reworks how exceptions are logged during bootstrap. To
address the concern of printing full exceptions to the console, a new
log4j exception filter is added to the console appender which will only
print out the exception message and some additional explanatory info. To
address logging multiple times, the try/catch within init is removed so
that excpetions can propagate to the try/catch in main, which now
handles all exceptions. Additionally, phase 1 (before logging) handles
it's own failure cases since there is definitely no logging at that
time. This simplifies the other failure cases latere so that they do not
need to check if logging has been initialized through sysprops.
The log4j configuration file is shipped with all ES distributions. We
also check recursively for files possibly added by plugins. If no files
are found, we give a helpful startup error message. However, since the
log4j2 configuration file shipped with ES should always exist, we can
check upfront in the cli before even initializing logging.
This commit moves the validation of an existing log4j2 properties file
to the server cli.
We don't build these libraries ourselves and the license forbids
us from modifying them in any way, so we won't be able to make
this rule pass on them. All we can do is override it.
Fixes#87632
The dataset value for all ES logs are prefixed with elasticsearch + log
type. Like elasticsearch.server. Deprecation log had it reverted
deprecation.elasticsearch
This commit renames the dataset for deprecation logs to
`elasticsearch.deprecation.
closes#83251
