The build handles platform specific code which may be for arm or x86.
Yet there are multiple ways to describe 64bit x86, and the build
converts between the two in several places. This commit consolidates on
the x64 nomenclature in most places, except where necessary (eg ML still
uses x86_64).
relates #105715
Elasticsearch requires access to some native functions. Historically
this has been achieved with the JNA library. However, JNA is a
complicated, magical library, and has caused various problems booting
Elasticsearch over the years. The new Java Foreign Function and Memory
API allows access to call native functions directly from Java. It also
has the advantage of tight integration with hotspot which can improve
performance of these functions (though performance of Elasticsearch's
native calls has never been much of an issue since they are mostly at
boot time).
This commit adds a new native lib that is internal to Elasticsearch. It
is built to use the foreign function api starting with Java 21, and
continue using JNA with Java versions below that.
Only one function, checking whether Elasticsearch is running as root, is
migrated. Future changes will migrate other native functions.
When hotspot encounters an error, it will emite a log file which can be
used for reproducing the error. This file is dumped to /tmp by default.
This commit configures the replay file to be alongside the hs_err file.
In certain circumstances if Elasticsearch encounters an error while
starting up, the server cli may exit with no error. This commit fixes
the cli to always check and wait on the Elasticsearch process and exit
with the same exit code.
relates #104055
Telemetry / APM settings are renamed from "tracing.apm.{name}" to "telemetry.tracing.{name}" for tracing related settings. General APM settings are renamed to "telemetry.{name}". The old legacy settings are kept for now and applied as fallback.
Prevent invalid and miss-configuration of APM agent using an explicit allow-list of setting keys.
Additionally, configuration defaults of APMAgentSettings are consolidated in APMJvmOptions to keep defaults in a single location.
(ES-6916)
We're leaking quite a few of these parsers. That doesn't seem to be much
of a problem but results in some memory inefficiencies in Jackson here
and there. This PR bulk fixes a bunch of instances that I could easily
automatically fix. I'll open a follow-up for closing the parser on the
document parsing context which also suffers from this but is non-trivial
to fix.
* Refactoring: move command line and environment build to a separate class
* Refactoring: adding a builder
* Moving tmp dir setup and JVM option parsing outside of builder
This commit upgrades the Bouncy Castle jars. Bouncy Castle is used for
some internal build concners as well as a comnand line application.
Most notably Bouncy Castle is also used as the FIPs certified JCE/JSEE provider
we use to test our ability to use a FIPs compliant crypto provider.
The following changes here are a result of the upgraded Bouncy Castle jars:
* TLSv1.3 is now supported when running in FIPs mode
* RSA PKCS#1 v1.5 is no longer allowed in FIPS mode
* Triple DES (3DES) is no longer allowed in FIPS mode
* Minor updates the security manager configuration used to test FIPs (to read permissions from the security provider)
* Minor adjustments to tests to accommodate the above changes.
* Minor adjustments to the gradle build to accommodate new dependencies
Note - update to the documentation will come in a later commit.
This introduces a new getHistoricalFeatures() method on ESRestTestCase
which returns a map of historical feature version mappings loaded from
FeatureSpecification implementations from any plugins/modules in use
by the current test suite. The mappings are generated by a new Gradle
task at build time, and then injected into the test runtime as a
System property.
* Make YamlRestTest and ElasticDistributionPlugin cc compatible
These changes are addressing configuration cache incompatibilities of running
rest tests with the new ClassRule based Elasticsearch test cluster setup.
This allows running those tests with configuration cache enabled:
`./gradlew :modules:ingest-user-info:yamlRestTest --configuration-cache`
node.processors determines the size of Elasticsearch threadpools. This
commit sets the JDK flag -XX:ActiveProcessorCount when node.processors
is set so that the JDK similarly sizes its threadpools accordingly.
relates #100244
* Make forbidden apis check cacheable and cc compatible
* Port CheckForbiddenApiTask to use worker api
* Simplify runtime classpath for CheckForbiddenApisTask
This commit forbids the use of Thread constructors that do not take a
name. In general nameless threads are more difficult to understand their
purpose when debugging. Note that this is only added to production
signatures. Tests are not forbidden here so as not to be pedantic (or
require a larger change since many tests create anonymous threads).
relates #101628
This commit makes the apm log a little more usable. First, it makes the
path to the log explicit by passing the ES logs dir through, instead of
relying on the location of the apm jar file. Second, it tweaks the log
level to be warn, not error. Third, it switches the apm log file to be
json, which is more easily processable.
The build version is made up of a few parts in non-release builds. Both
the snapshot and pre-release qualifiers are appended to it. These
qualifiers used to be part of Version, but in 7.0 the qualifiers were
made to be found only in the build info. The Build class retains these
qualifiers through the compile ES version extracted from the server jar
at runtime.
Build.qualifiedVersion() is suppose to provide the fully qualified
version, including snapshot and pre-release qualifiers. Yet
Build.version() also includes this information; there is no distinction
since the qualifier was moved to be only in the build info.
This commit separates the pre-release qualifier from the version. It
maintains bwc in talking to older nodes, passing the fully qualified
version there, but in current nodes splits out the pre-release qualifier
into a new member of Build.
The Plugin CLI can generally treat strings opaquely. We had some logic
comparing earlier versions, but what we really care about with most of our
plugins is whether or not they were built with the current version of
Elasticsearch, not whether they were built before or after. (This question
will be trickier with stable plugins, but none of that code is in the CLI.)
The CLI classes can be cleaned up even more once Version is removed from
PluginDescriptor. Some of the tests can't use opaque strings for versions
until PluginDescriptor can handle them.
* Remove Version from Install and List plugin actions
* Remove Version from SyncPluginsAction
Another round of automated fixes to this, marking things that can be
made static as static. Saves some JIT cycles but also turns some lambdas
from capturing to non-capturing and makes the "utilityness" of some
classes visible.
There is a bug in the nebula os package gradle plugin that breaks copy spec specific setgid handling.
We have created a patch for the plugin that we use for now to unblock us. This will be ported upstream
to the nebula main branch and part of a release but that requires some more polishing and will
be taken care of in a later PR
The latest version contains a fix to allow sending metrics to APM server. also adds a apm agent jvm options
"enable_experimental_instrumentations", "true"
which is required to enable the otel-metrics-instrumentation.
relates https://github.com/elastic/elasticsearch/pull/99832
* Replaced some occurrences of Version.CURRENT toString
* Added unit tests for the 2 different warn header patterns in HeaderWarnings
* Comment to clarify non-semantic versions + revert change on Version.major message
Gradle now fully supports compiling, testing and running on Java 20.
Among other general performance improvements this release introduces --test-dry-run command line option that allows checking if tests are filtered or not by gradle.
Required updating nebula ospackage plugin as setuid was broken in gradle 8.3.
This commit updates the version range in jvm.options for the Panama Vector API. The range is updated from just a single value 20, to a range starting from 20 with no upper bound. This effectively enables the Panama Vector API for use in Lucene on all JDK versions greater than 19.
with the support of metrics the TracerPlugin name is no longer adequate. Renaming this to TelemetryPlugin.
Also introducing TelemetryProvider interface. While it is only used in Node.java at the moment to fetch Tracer instance, it is intended to be used in Plugin::createComponents (to be done in separate commit due to
the broad scope of this method)
This will allow for plugins to get access to both Tracer and Metric interfaces
without the need to add yet another argument to createComponents
Also adding internal subpackage in module/apm so that it is more obvious
which packages are not exported
Elasticsearch use lib as its module path, which includes all the
dependencies of server. However, there are sometimes additional modules
such as preallocate that are used dynamically and not direct
dependencies of server. This commit switches the addition of these
modules to be automatic by including all modules present in the module
path.
In order for build info to be pluggable for serverless, the current
build needs to be lazily determined. This commit moves the CURRENT
constant to a static method.
relates #96861
