* Add support for building against defined build-ids of dependent artifacts
* Introduce DraResolvePlugin and functional tests
* Declare initial dra snapshot ci job for elasticsearch
We run the DRA builds against ubuntu as other release related ci jobs have in the past. Also we only have libs2xmlutills available on our ubuntu based ci workers
When we launch Elasticsearch with the APM monitoring
agent, we create a temporary configuration file to
securely pass the API key or secret. This temporary
file is cleaned up on Elasticsearch Node creation.
After we renamed the APM module, the delete logic
didn't get updated, which means we never delete the file
anymore.
This commit:
- fixes the APM module pattern match when we delete
- adds additional delete safety net on failed node start
- adds tests for ensuring the naming dependency isn't
broken again.
Removing the custom dependency checksum functionality in favor of Gradle build-in dependency verification support.
- Use sha256 in favor of sha1 as sha1 is not considered safe these days.
Closes https://github.com/elastic/elasticsearch/issues/69736
Part of #84369. Implement the `Tracer` interface by providing a
module that uses OpenTelemetry, along with Elastic's APM
agent for Java.
See the file `TRACING.md` for background on the changes and the
reasoning for some of the implementation decisions.
The configuration mechanism is the most fiddly part of this PR. The
Security Manager permissions required by the APM Java agent make
it prohibitive to start an agent from within Elasticsearch
programmatically, so it must be configured when the ES JVM starts.
That means that the startup CLI needs to assemble the required JVM
options.
To complicate matters further, the APM agent needs a secret token
in order to ship traces to the APM server. We can't use Java system
properties to configure this, since otherwise the secret will be
readable to all code in Elasticsearch. It therefore has to be
configured in a dedicated config file. This in itself is awkward,
since we don't want to leave secrets in config files. Therefore,
we pull the APM secret token from the keystore, write it to a config
file, then delete the config file after ES starts.
There's a further issue with the config file. Any options we set
in the APM agent config file cannot later be reconfigured via system
properties, so we need to make sure that only "static" configuration
goes into the config file.
I generated most of the files under `qa/apm` using an APM test
utility (I can't remember which one now, unfortunately). The goal
is to setup up a complete system so that traces can be captured in
APM server, and the results in Elasticsearch inspected.
The new stable plugin api will have a slightly different descriptor file
format. This commit prepares for stable plugins by adding support for
reading those new files. The basic info for a plugin is the same like
name and version info. Other stuff like classname are not necessary. The
one additional property specific to the new plugins is "modular", which
indicates whether the jars of the plugin should be loaded as named
modules (this is akin to setting the module path when running java).
As part of #50277, we removed the `TAKE_FILE_OWNERSHIP` option from the
Docker entrypoint script and the associated chroot calls, and instead
just defaulted to running the image as `elasticsearch` instead of
`root`.
However, we didn't check that it was still possible to pass CLI options
to Elasticsearch via CLI arguments, and broke this by mistake. This is
probably an uncommon pattern, versus environment variables or a config
file. Nevertheless, it is supposed to be possible and is mentioned in
the documentation.
Fix the problem by suppling the missing positional params when calling
Elasticsearch, and add a test case so that we don't break it again.
As part of #50277, we removed the `TAKE_FILE_OWNERSHIP` option from the
Docker entrypoint script and the associated chroot calls, and instead
just defaulted to running the image as `elasticsearch` instead of
`root`.
However, we didn't check that it was still possible to pass CLI options
to Elasticsearch via CLI arguments, and broke this by mistake. This is
probably an uncommon pattern, versus environment variables or a config
file. Nevertheless, it is supposed to be possible and is mentioned in
the documentation.
Fix the problem by suppling the missing positional params when calling
Elasticsearch, and add a test case so that we don't break it again.
This changes the LoggedExec task to be configuration cache compatible. We changed the implementation
to use `ExecOperations` instead of extending `Exec` task. As double checked with the Gradle team this task
is not planned to be made configuration cache compatible out of the box anytime soon.
This is part of the effort on https://github.com/elastic/elasticsearch/issues/57918
This adds the generation and upload logic of Gradle dependency graphs to snyk
We directly implemented a rest api based snyk plugin as:
the existing snyk gradle plugin delegates to the snyk command line tool the command line tool
uses custom gradle logic by injecting a init file that is
a) using deprecated build logic which we definitely want to avoid
b) uses gradle api we avoid like eager task creation.
Shipping this as a internal gradle plugin gives us the most flexibility as we only want to monitor
production code for now we apply this plugin as part of the elasticsearch.build plugin,
that usage has been for now the de-facto indicator if a project is considered a "production" project
that ends up in our distribution or public maven repositories. This isnt yet ideal and we will revisit
the distinction between production and non production code / projects in a separate effort.
As part of this effort we added the elasticsearch.build plugin to more projects that actually end up
in the distribution. To unblock us on this we for now disabled a few check tasks that started failing by applying elasticsearch.build.
Addresses #87620
The ingest attachment processor is currently available as a plugin. This
commit moves the processor to the default distribution so it is always
available.
Bootstrap plugins were an internal mechanism added to allow a
filesystemprovider for cloud with the quota-aware-fs plugin. Since that
was removed, bootstrap plugins no longer serve a purpose. They were
never officially documented because they were for internal use only.
This commit removes the bootstrap plugins infrastructure.
The keystore format has been changed a few times since it was first
introduced. Part of Elasticsearch startup automatically upgrades the
format. Since Elasticsearch has fixed bounds of supported versions for
upgrades, there are also fixed bounds on the keystore formats we might
need to read.
The v3 keystore format was introduced in Elasticsearch 6.3.0. Since
current Elasticsearch master branch is 8.x, and 8.x only supports
offline upgrades from 7.x, it is therefore impossible to need to read
v1 or v2 formats. This commit removes support for those formats.
During startup of Elasticsearch we go to great lengths to present errors
in a meaningful way to users. Over time, though, the error handling has
been amended to address various issues, and that has resulted in a
complicated system of try/catches handling various cases. One
particularly kludgy piece is removing the console logger in special
cases to avoid printing exceptions to the console. Additionally, the
console removal wasn't actually effective because later in exception
handling the exception would be both logged anyways, and then also sent
to stderr, meaning that we could see the same exception several times.
This commit reworks how exceptions are logged during bootstrap. To
address the concern of printing full exceptions to the console, a new
log4j exception filter is added to the console appender which will only
print out the exception message and some additional explanatory info. To
address logging multiple times, the try/catch within init is removed so
that excpetions can propagate to the try/catch in main, which now
handles all exceptions. Additionally, phase 1 (before logging) handles
it's own failure cases since there is definitely no logging at that
time. This simplifies the other failure cases latere so that they do not
need to check if logging has been initialized through sysprops.
The log4j configuration file is shipped with all ES distributions. We
also check recursively for files possibly added by plugins. If no files
are found, we give a helpful startup error message. However, since the
log4j2 configuration file shipped with ES should always exist, we can
check upfront in the cli before even initializing logging.
This commit moves the validation of an existing log4j2 properties file
to the server cli.
We don't build these libraries ourselves and the license forbids
us from modifying them in any way, so we won't be able to make
this rule pass on them. All we can do is override it.
Fixes#87632
The dataset value for all ES logs are prefixed with elasticsearch + log
type. Like elasticsearch.server. Deprecation log had it reverted
deprecation.elasticsearch
This commit renames the dataset for deprecation logs to
`elasticsearch.deprecation.
closes#83251
When running in Docker, the elasticsearch-plugins.yml allows configuring
plugins that should be installed in the system. Upon Elasticsearch
starting up, plugins are installed/removed to match the configured
plugins. However, this happens late in startup, and it would be nice to
keep the main Elasticsearch process from ever writing outside the
configured data directories. Now that the server cli has been moved to
Java, this is possible.
This commit moves invocation of the plugins sync command into the server
cli. Note that the sync plugins action should probably be reworked as it
can be implement Command directly now. However, this commit tries to be
the minimal change possible to remove plugin cli knowledge from server.
When running Elasticsearch in the foreground, the cli process waits
indefinitely on the server. If the server dies unexpectedly, the
ServerProcess throws an exception. However, the exit code is hidden in
the exception message. This commit changes waitFor to return the exit
code, so it can be propagated to the cli main. Note that when stopping
in a shutdown hook the exit code must be ignored because calling
System.exit from a shutdownhook results in a deadlock.
When the server process dies, the cli should reflect the unexpected exit
of the process in its return from waitFor. The test for this closed the
input side of the pipe, but that is where the buffer lives. This means
there is a race condition in the cli testcase fully reading the final
message expected. This commit removes some complexity from the test by
using a single latch, and closing the pipe from the process side to
mimic the break.
The class PluginInfo represents the plugin-descriptor.properties file
that each plugin must have. This commit renames the class to more
closely match what it represents: the plugin descriptor.
This PR represents the initial phase of Modularizing Elasticsearch (with
Java Modules).
This initial phase modularizes the core of the Elasticsearch server
with Java Modules, which is then used to load and configure extension
components atop the server. Only a subset of extension components are
modularized at this stage (other components come in a later phase).
Components are loaded dynamically at runtime with custom class loaders
(same as is currently done). Components with a module-info.class are
defined to a module layer.
This architecture is somewhat akin to the Modular JDK, where
applications run on the classpath. In the analogy, the Elasticsearch
server modules are the platform (thus are always resolved and present),
while components without a module-info.class are non-modular code
running atop the Elasticsearch server modules. The extension components
cannot access types from non-exported packages of the server modules, in
the same way that classpath applications cannot access types from
non-exported packages of modules from the JDK. Broadly, the core
Elasticseach java modules simply "wrap" the existing packages and export
them. There are opportunites to export less, which is best done in more
narrowly focused follow-up PRs.
The Elasticsearch distribution startup scripts are updated to put jars
on the module path (the class path is empty), so the distribution will
run the core of the server as java modules. A number of key components
have been retrofitted with module-info.java's too, and the remaining
components can follow later. Unit and functional tests run as
non-modular (since they commonly require package-private access), while
higher-level integration tests, that run the distribution, run as
modular.
Co-authored-by: Chris Hegarty <christopher.hegarty@elastic.co>
Co-authored-by: Ryan Ernst <ryan@iernst.net>
Co-authored-by: Rene Groeschke <rene@elastic.co>
On Windows cmd, one can invoke `myscript.bat` by just typing `myscript`.
This commit adjusts the CliToolLauncher to account for this case, making
the .bat stripping to find the toolname conditional.
closes#86940
Now that the server cli is in java, we can do more system level things
inside it. This commit moves validating and writing the pidfile into the
server cli. One benefit is we get validation of directory/file problems
up front before even trying to start the ES process.
