We see an certificate issue when using fips docker image creating searchable snapshots in aws
and gcs. This is likely related to a configuration issue not explicitly setting the trust store
type for our bcfks cacerts
* Delegated authorization using Microsoft Graph (SDK)
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Johannes Freden Jansson <johannes.freden@elastic.co>
Co-authored-by: Johannes Fredén <109296772+jfreden@users.noreply.github.com>
* Include direct memory and non-heap memory in ML memory calculations.
* Reduce ML_ONLY heap size, so that direct memory is accounted for.
* [CI] Auto commit changes from spotless
* changelog
* improve docs
* Reuse direct memory to heap factor
* feature flag
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Restructures docker files for docker distributions
- Put Dockerfiles in specific distro specific folders keeping "Dockerfile" naming convention
- Allows better ide support
- Allows easier renovate integration
- Explicitly set base image in dockerfile
- simplify renovate configuration
- Cleanup DockerBase file to not contain ess fips base image information
This lives now in the Dockerfile content directly
* Workaround docker test issue
* Fix labels for fips image
This change enhances the dense_vector section of the Nodes stats and Index stats APIs so that they report the desired size of off-heap memory for all indexed vectors. The dense_vector section of the Custer stats API remains unchanged.
The retrieval mechanism and structure of the new stats is the same across the various three stats APIs, but more fine-grained information is disclosed as when moving from Cluster -> Node -> Index API.
For Node stats, we aggregate the total byte sizes for all vectors, categorised by the data type. For example:
"dense_vector" : {
"value_count" : 5,
"off_heap" : {
"total_size_in_bytes" : 27,
"total_veb_size_in_bytes" : 3,
"total_vec_size_in_bytes" : 23,
"total_veq_size_in_bytes" : 0,
"total_vex_size_in_bytes" : 1
}
}
Index stats: same as Node stats with included field break down . For example:
"dense_vector" : {
"value_count" : 5,
"off_heap" : {
"total_size_in_bytes" : 27,
"total_veb_size_in_bytes" : 3,
"total_vec_size_in_bytes" : 23,
"total_veq_size_in_bytes" : 0,
"total_vex_size_in_bytes" : 1,
"fielddata" : {
"bar" : {
"veb_size_in_bytes" : 3,
"vec_size_in_bytes" : 14,
"vex_size_in_bytes" : 1
},
"foo" : {
"vec_size_in_bytes" : 9
}
}
}
The implementation accesses the actual statistics through reflection. This will be completely removed when Lucene exposes this, which is expected in Lucene 10.3
Now that entitlements are always used, there is no need to run tests
with security manager (a future enhancement will run tests with
entitlements). This commit removes setting up security manager from
tests.
The add string keystore command accepts input on stdin. In this case,
the terminal is responsible for separating input by newlines that are
system dependent. The mock terminal normally adds the system dependent
newline. However, we also have tests that ensure both unix and windows
style newlines are tested explicitly. This commit fixes the windows
style carriage returns to explicitly contain a newline.
closes#127089
As a followon to #126729, the add string keystore command doesn't need
to use a reader at all (and it was incorrect for it to close the reader
from the terminal). Instead, the Terminal abstraction already handles
how to get at line by line secrets. This commit removes that usage of
reader and uses readSecret calls instead.
closes#126882
With the change to using the logs dir as the working dir of the
Elasticsearch process we need to ensure the logs dir exists within the
CLI instead of later during startup.
relates #124966
When reading a string value from stdin the keystore add command
currently looks directly at stdin. However, stdin may also be consumed
while reading the keystore password. This commit changes the add command
to use the reader from the termainl instead of looking at stdin
directly.
closes#98115
This addresses feedback we got for our default image at https://github.com/docker-library/official-images/pull/18692
This also introduces separate docker source files to make maintaining those easier.
We cannot take over all suggested changes as we require certain settings to have our packaging tests pass as expected.
In the unexpected case that Elasticsearch dies due to a segfault or
other similar native issue, a core dump is useful in diagnosing the
problem. Yet core dumps are written to the working directory, which is
read-only for most installations of Elasticsearch. This commit changes
the working directory to the logs dir which should always be writeable.
This PR replaces the parsing and formatting of SecurityManager policies with the parsing and formatting of Entitlements policy during plugin installation.
Relates to ES-10923
- Adds docker image based on chainguard base fips image
- x86 only for now as the base image is x86 only
- the image does not provide any elasticsearch.yml configuration. for testing purposes you can follow the elasticsearch fips guide available at https://github.com/elastic/FIPSGuide/tree/main/elasticsearch
The image is shipped with:
- org.bouncycastle:bc-fips:1.0.2.5 and org.bouncycastle:bctls-fips:1.0.19 in Elasticsearch libs folder
- config/jvm.options.d/fips.options for fips specific JVM options
- fips_java.security file
- fips_java.policy
Out of scope:
- Add packaging test coverage (part of later PR as we want to provide that image for testing early and packaging tests require more general restructuring for support fips scenarios)
This PR adds a check and a warning message if the user is trying to install a plugin containing the legacy plugin-security.policy used by SecurityManager.
Relates to ES-10846
The JDK team has completely disabled the Java SecurityManager from Java 24. Elasticsearch has always used the Java SecurityManager as an additional protection mechanism; in order to retain this second line of defense, the Elasticsearch Core/Infra team has been working on the Entitlements project.
Similar to SecurityManager, Entitlements only allow calling specific methods in the JDK when the caller has a matching policy attached. In other words, if some code (in the main Elasticsearch codebase, in a plugin/module, or in a script) attempts to perform a "privileged" operation and it is not entitled to do so, a NotEntitledException will be thrown.
This PR includes the minimal set of changes to always use Entitlements, regardless of system properties or Java version.
Relates to ES-10921
Information about the system, and in particular the running JVM, is
logged relatively late in startup. When problems occur in low level
initialization, it is often useful to have this information. This commit
moves logging of system information to right after logging has been
initialized. Note that it no longer utilizes the JvmInfo class as most
was already easily available and the static init of JvmInfo does a lot.
The JVM has a couple ways that Java options can be passed. In
Elasticsearch we ignore these common patterns, instead using our own
bespoke mechanisms for setting options (ES_JAVA_OPTS, etc). The primary
reason for this is some systems like Ubuntu setting these options
globally and then affecting Elasticsearch, often in ways that cause
Elasticsearch to fail to start.
This commit ignores _JAVA_OPTIONS, which is a little known environment
variable that the JVM may pick up options from.
By default Lucene only uses vectorization up to the JDK version
available at the time of its release. When testing against EA versions,
we want to still use vectorization. This commit adds a sysprop to force
lucene to use the current version for vectorization.
This commit fix the IronBank hardening_manifest CI test that expect the
hardening_manifest.yaml values to match `'^(?!\\s*$)[ -~|]*$'` regex:
```
| hardening_manifest | ERROR | Hardening Manifest failed jsonschema validation
| hardening_manifest | ERROR | Verify Hardening Manifest content
| hardening_manifest | ERROR | 'Rene Gröschke' does not match '^(?!\\s*$)[ -~|]*$'
```
43219286
cc @breskeby
<!-- Thank you for your interest in and contributing to Elasticsearch!
There are a few simple things to check before submitting your pull
request that can help with the review process. You should delete these
items from your submission, but they are here to help bring them to your
attention. -->
- Have you signed the [contributor license agreement](https://www.elastic.co/contributor-agreement)?
- Have you followed the [contributor guidelines](https://github.com/elastic/elasticsearch/blob/main/CONTRIBUTING.md)?
- If submitting code, have you built your formula locally prior to submission with `gradle check`?
- If submitting code, is your pull request against main? Unless there is a good reason otherwise, we prefer pull requests against main and will backport as needed.
- If submitting code, have you checked that your submission is for an [OS and architecture that we support](https://www.elastic.co/support/matrix#show_os)?
- If you are submitting this code for a class then read our [policy](https://github.com/elastic/elasticsearch/blob/main/CONTRIBUTING.md#contributing-as-part-of-a-class) for that.
* Fix Gradle Deprecation warning as declaring an is- property with a Boolean type has been deprecated.
* Make use of new layout.settingsFolder api to address some cross project references
* Fix buildParams snapshot check for multiprojet projects
* Remove @UpdateForV9 annotation in RuntimeVersionFeature
Re ES-10338 the Core/Infra team agreed that this is not blocked on security manager removal. We should go ahead and remove these 2 instances of the @UpdateForV9 annotation, the code will be cleaned up soon with the removal of Security Manager.
* Remove @UpdateForV9 annotation from SystemJvmOptions
re ES-10338 the Core/Infra team agreed this is not blocked on security manager removal. We should go ahead and remove this instance of the @UpdateForV9 annotation.
