This reverts #117106. Bwc tests fail, because older nodes are killed with the following error:
```
[2024-11-20T10:54:58,600][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [v8.17.0-0] fatal error in thread [elasticsearch[v8.17.0-0
][clusterApplierService#updateTask][T#1]], exiting java.lang.AssertionError: provided source [{"_doc":{"_data_stream_timestamp":{"enabled":true},"_source":{},"properties":{"@timestamp":{"type":"date"},"k8s":{"properties":{"pod":{"properties":{"ip":{"type":"ip"},"name":{"type":"keyword"},"network":{"properties":{"rx":{"type":"long"},"tx":{"type":"long"}}},"uid":{"type":"keyword","time_series_dimension":true}}}}},"metricset":{"type":"keyword","time_series_dimension":true}}}}] differs from mapping [{"_doc":{"_data_stream_timestamp":{"enabled":true},"_source":{"mode":"synthetic"},"properties":{"@timestamp":{"type":"date"},"k8s":{"properties":{"pod":{"properties":{"ip":{"type":"ip"},"name":{"type":"keyword"},"network":{"properties":{"rx":{"type":"long"},"tx":{"type":"long"}}},"uid":{"type":"keyword","time_series_dimension":true}}}}},"metricset":{"type":"keyword","time_series_dimension":true}}}}]
at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.index.mapper.DocumentMapper.<init>(DocumentMapper.java:66)
at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.index.mapper.MapperService.newDocumentMapper(MapperService.java:588)
at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.index.mapper.MapperService.updateMapping(MapperService.java:346)
at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.index.IndexService.updateMapping(IndexService.java:840)
at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.indices.cluster.IndicesClusterStateService.createIndicesAndUpdateShards(IndicesClusterStateService.java:583)
at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.indices.cluster.IndicesClusterStateService.doApplyClusterState(IndicesClusterStateService.java:306)
at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:260)
at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:544)
at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:530)
at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:503)
at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:432)
at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:157)
at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:956)
at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:218)
at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:184)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1575)
```
The `mode` parameter no longer gets serialized for new indices. However on the older nodes still serialize the `mode` parameter, which caused the menioned assertion to fail. Reverting for now and see how best to address this bwc serialization issue.
We can only stop serializing mode, when all nodes are on the same version. Unfortunately we can't invoke `c.clusterTransportVersion().get()` from parser or builder, because that calling thread isn't allowed to call `clusterService.state()`.
* Add initial entitlement policy parsing (#114448)
This change adds entitlement policy parsing with the following design:
* YAML file for readability and re-use of our x-content parsers
* hierarchical structure to group entitlements under a single scope
* no general entitlements without a scope or for the entire project
* Avoid double instrumentation via class annotation (#115398)
* Move entitlement jars to libs (#115883)
The distribution tools are meant to be CLIs. This commit moves the
entitlements jar projects to the libs dir, under a single
libs/entitlement root directory to keep the related jars together.
* Entitlement tools: SecurityManager scanner (#116020)
* Dynamic entitlement agent (#116125)
* Refactor: treat "maybe" JVM options uniformly
* WIP
* Get entitlement running with bridge all the way through, with qualified
exports
* Cosmetic changes to SystemJvmOptions
* Disable entitlements by default
* Bridge module comments
* Fixup forbidden APIs
* spotless
* Rename EntitlementChecker
* Fixup InstrumenterTests
* exclude recursive dep
* Fix some compliance stuff
* Rename asm-provider
* Stop using bridge in InstrumenterTests
* Generalize readme for asm-provider
* InstrumenterTests doesn't need EntitlementCheckerHandle
* Better javadoc
* Call parseBoolean
* Add entitlement to internal module list
* Docs as requested by Lorenzo
* Changes from Jack
* Rename ElasticsearchEntitlementChecker
* Remove logging javadoc
* exportInitializationToAgent should reference EntitlementInitialization, not EntitlementBootstrap.
They're currently in the same module, but if that ever changes, this code would have become wrong.
* Some suggestions from Mark
---------
Co-authored-by: Ryan Ernst <ryan@iernst.net>
* Remove unused EntitlementInternals (#116473)
* Revert "Entitlement tools: SecurityManager scanner (#116020)"
This reverts commit 023fb663de.
---------
Co-authored-by: Jack Conradson <osjdconrad@gmail.com>
Co-authored-by: Lorenzo Dematté <lorenzo.dematte@elastic.co>
Co-authored-by: Ryan Ernst <ryan@iernst.net>
* [TEST] Migrated REST tests extending JsonLogsIntegTestCase (#115188)
REST tests extending JsonLogsIntegTestCase migrated to the new REST
testing framework, using 'elasticsearch.internal-java-rest-test' Gradle
plugin
Explicit handling of test single-cluster config by checking if cluster
is explicitly configured as single-node to avoid an incorrect configuration
which may cause split brain.
Replaced `ElasticsearchJavaPlugin` with `ElasticsearchJavaBasePlugin` in
`RestTestBasePlugin` for better granularity.
Additionally, updated the `DistributionDownloadPlugin` to set configurations as
non-consumable and the `InternalDistributionArchiveSetupPlugin` to mark
certain configurations as consumable. This ensures that configurations
are correctly utilized during build setup and execution phases.
(cherry picked from commit 130cc74d8a)
* Fixed DefaultSettingsProvider to use Java 17 (#116029)
It is needed to be able to backport the change to the 8.x.
Having same code for both versions can help us to avoid some merge
conflicts in the future.
Backport #115639 to 8.x branch.
The main difference between other rolling upgrade tests is that these tests index more data while performing the rolling upgrade and no rollover is performed during rolling upgrade. For example this makes it more likely for merging to happen, which could uncover bwc bugs.
Note that currently both test suites start trial license so that synthetic source gets used.
* Use directory name as project name for libs (#115720)
The libs projects are configured to all begin with `elasticsearch-`.
While this is desireable for the artifacts to contain this consistent
prefix, it means the project names don't match up with their
directories. Additionally, it creates complexities for subproject naming
that must be manually adjusted.
This commit adjusts the project names for those under libs to be their
directory names. The resulting artifacts for these libs are kept the
same, all beginning with `elasticsearch-`.
* fixes
* Replace cloud-ess docker image with wolfi-ess
We just replaced the existing implementation of cloud-ess with what was wolfi-ess which is a wolfi based ess image.
The cloud image itself will be removed in a future commit it was not used anywhere
* Switch to test cloud docker image instead of default docker in packaging pr tests.
This adds way more coverage than the default docker image which is also barely touched
(cherry picked from commit f1f5ee06a3)
# Conflicts:
# build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/DockerBase.java
This removes the undocumented `META FUNCTIONS` command that emits
descriptions for all functions. This shouldn't be used because we never
told anyone about it. I'd have preferred if we'd have explicitly
documented it as no public or if we'd have left it snapshot-only. But
sometimes you make a mistake. I'm hopeful no one is relying on it. It
was never reliable and not public.....
Updating `com.google.protobuf:protobuf-java-util` and
`com.google.protobuf:protobuf-java` dependencies to
the latest 3.x version.
(cherry picked from commit ddf019a1a5)
The option is only enabled for test clusters, and turns on features that are only present in tests
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Extensible plugins use a custom classloader for other plugin jars. When
extensible plugins were first added, the transport client still existed,
and elasticsearch plugins did not exist in the transport client (at
least not the ones that create classloaders). Yet the transport client
still created a PluginsService. An indirection was used to avoid
creating separate classloaders when the transport client had created the
PluginsService.
The transport client was removed in 8.0, but the indirection still
exists. This commit removes that indirection layer.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Remove zstd feature flag for index codec best compression. (#112665)
ZStandard was added via #103374 a few months ago to snapshot builds of Elasticsearch only and benchmark results have shown that using zstd is a better trade off compared to deflate for when index.codec is set to best_compression.
This change removes the feature flag for ZStandard stored field compression for indices with index.codec set to best_compression.
* Update docs/changelog/112857.yaml
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
JDK 23 removes the COMPAT locale provider, leaving CLDR as the only option. This commit configures Elasticsearch
to use the CLDR provider when on JDK 23, but still use the existing COMPAT provider when on JDK 22 and below.
This causes some differences in locale behaviour; this also adapts various tests to still work whether run on COMPAT or CLDR.
Native libraries in Java are loaded by calling System.loadLibrary. This
method inspects paths in the java.library.path to find the requested
library. Elasticsearch previously used this to find libsystemd, but now
the only remaining use is to set the additional platform directory in
which Elasticsearch keeps its own native libraries.
One issue with setting java.library.path is that its not set for the cli
process, which makes loading the native library infrastructure from clis
difficult. This commit reworks how Elasticsearch native libraries are
found in order to avoid needing to set java.library.path. There are two
cases. The simplest is production, where the working directory is the
Elasticsearch installation directory, so the platform specific directory
can be constructed. The second case is for tests where we don't have an
installtion. We already pass in java.library.path there, so this change
renames the system property to be a test specific property that the new
loading infrastructure looks for.
