JDK 22 may return a console even if the terminal is redirected. These cases are detected using the new Console#isTerminal() to maintain the current behavior (closes#98033).
Co-authored-by: Moritz Mack <mmack@apache.org>
* Update Gradle Wrapper to 8.2 (#96686)
- Convention usage has been deprecated and was fixed in our build files
- Fix test dependencies and deprecation
Improve InnocuousThread permission checks handling (6a3855112c)
on shutdown, the jdk's InnocuousThread can try to change a thread name. This requires "java.lang.RuntimePermission" "modifyThread" permission. However InnocuousThread doe not inherit any Access Control Context and therefore have no permissions. This results in AccessControlException.
This commit fixes this by skipping a check for modify thread permission if a thread is innocuous.
relates #91658 and #91650
When previously described AccessControlException is thrown, it is not being catched anywhere in the Elasticsearch, hence it ends up being handled by ElasticsearchUncaughtExceptionHandler#onNonFatalUncaught
This is being again being run by the thread [process reaper] which is an innocuous thread (jdk specific) and has no permissions. onNonFatalUncaught is trying to log a message, but this in turn requires java.lang.RuntimePermission" "getenv." permission. which is does not have. This again results in AccessControlException java.lang.RuntimePermission" "getenv."
We can fix this by executing with doPrivileged in ElasticsearchUncaughtExceptionHandler#onNonFatalUncaught and this will stop the Security Manager's walk and will have ES's global grant permissions.
backport(#91704)
closes#91650
Update the ES security manager to tolerate the creation of the JDK's InnocuousThreads, to facilitate the use of java.lang.ref.Cleaner in ES and dependent code by default. Further details in issue ( #77788 )
resolves#77788
backport note - in 7.17 branch jdk8 is still supported. Therefore in order to recognize jdk.base domain it has to check for classloader == null (meaning it is a bootstrap classloader) instead of comparing modules
backport(#77788)
Backports the following commits to 7.17:
- Add verification metadata for dependencies (#88814)
- Add missing dependency verification checksums (#89139)
Closes#70219.
Introduce a declarative way for the Elasticsearch server to manage plugins,
which reads the `elasticsearch-plugins.yml` file and works which out
plugins need to be added and / or removed to match the configuration. Also
make it possible to configure a proxy in the config file, instead of
through the environment.
Most of the work of adding and removing is still done in the
`InstallPluginAction` and `RemovePluginAction` classes, so the
behaviour should be the same as with the `install` and `remove`
commands. However, these commands will now abort if the above config
file exists. The intent is to make it harder for the configuration
to drift.
This new method only applies to `docker` distribution types at the
moment.
Since this syncing mechanism declarative, rather than imperative,
the Cloud-specific plugin wrapper script is no longer required.
Instead, an environment variable informs `InstallPluginAction` to
install plugins from an archive directory instead of downloading
them, where possible.
Mockito 4.0 removes several deprecated methods. This commit updates
usages of those deprecated methods and upgrades mockito. The changes
include: * Replace anyMapOf,anyListOf,anySetOf,anyCollectionOf with the
same method name without `Of` and no longer taking any arguments. *
Replace anyObject with any * Removing argument from isNull * Replace
verifyZeroInteractions with verifyNoMoreInteractions The changes here
were completely mechanical, done entirely with forms of find/replace
within IntelliJ.
Matchers is deprecated in Mockito, in favor of the newer
ArgumentMatchers class. In fact, internally Matchers just extends
ArgumentMatchers as all the methods there were moved. This commit
changes all imports of org.mockito.Matchers to
org.mockito.ArgumentMatchers.
This commit adds support for decrypting PKCS#8 encoded private keys
that have been encrypted using a PBES2 based scheme (AES only).
Unfortunately `java.crypto.EncryptedPrivateKeyInfo` doesn't make this
easy as the underlying encryption algorithm is hidden within the
`AlgorithmParameters`, and can only be extracted by calling
`toString()` on the parameters object.
See: https://datatracker.ietf.org/doc/html/rfc8018#appendix-A.4
See: AlgorithmParameters#toString()
See: com.sun.crypto.provider.PBES2Parameters#toString()
This support is conditional on the underlying support in the JDK, which is absent from OpenJDK 8, but should work on all other supported JDKs.
Backport of: #78904
Securemock is a wrapper around Mockito that monkey patches internals of
Mockito to work with the SecurityManager. However, the library has not
been updated in several years due to the complicated nature of this
monkey patching. This has left us with an ancient version of Mockito,
missing out on updates to the library in the last half decade.
While Securemock currently works with Mockito 1.x, in 2.x an official
means of plugging into mockito was added, MockMaker. This commit removes
securemock as a dependnecy of the test framework, replacing it with a
modern version of Mockito, and implementing a MockMaker that integrates
with SecurityManager.
Note that while there is a newer version of Mockito available, 4.0, it
has several deprecations removed that are used throughout Elasticsearch.
Those can be addressed in followups, and then a subsequent upgrade to
4.0 should be possible.
relates #79567closes#40334
Optimize object parsers a little by extracting cold paths, removing some unnecessary
lambda wrapping and some other small things.
Also, fixed a very expensive use of these APIs in Phase moving from a very hot stream
instantiation to a standard loop.
A recent change for the deprecation logs provided the capability to emit deprecation's at critical vs. warning levels, #77482.
However deprecated settings always log at critical level without the ability to express that the setting deprecation is only a
warning.
This commit exposes the ability to set the deprecation level when deprecating a setting.
Relates #78781
* Replace eager gradle task creation with task avoidance api
Some more eagerly created tasks sneaked into the 7.x branch lately
* Fix typo in integTest configuration
* Move xcontent filtering tests (#79298)
* Move xcontent filtering tests
Moves the xcontent filtering tests to the xcontent project because its
testing code *in* the xcontent project.
* More clear
* Spotless
* Fixup
Java9 added a number of features that are useful to improve compression
and decompression. These include the Arrays#mismatch method and
VarHandles. This commit adds compression tools forked from the java-lz4
library which include these improvements. We hope to contribute these
changes back to the original project, however the project currently
supports Java7 so this is not possible at the moment.
Found this when benchmarking large cluster states. When serializing collections we'd mostly
not take any advantage of what we know about the collection contents (like we do in `StreamOutput`).
This PR adds a couple of helpers to the x-content-builder similar to what we have on `StreamOutput`
to allow for faster serializing by avoiding the writer lookup and some self-reference checks.
This adds a new "elasticsearch-keystore show" command that displays
the value of a single secure setting from the keystore.
Additionally, support for direct (stream) output has been added to
the Terminal class so that non-UTF8 values can be written to
stdout, and accordingly this command works correctly:
elasticsearch-keystore show xpack.watcher.encryption_key > watcher.key
Backport of: #76693
Co-authored-by: Ioannis Kakavas <ikakavas@protonmail.com>
Co-authored-by: Ioannis Kakavas <ikakavas@protonmail.com>
Today we open connections to other nodes in various places and largely
assume that they remain open as needed, only closing them when applying
a cluster state that removes the remote node from the cluster. This
isn't ideal: we might preserve unnecessary connections to remote nodes
that aren't in the cluster if they never manage to join the cluster, and
we might also disconnect from a node that left the cluster while it's in
the process of re-joining too (see #67873).
With this commit we move to a model in which each user of a connection
to a remote node acquires a reference to the connection that must be
released once it's no longer needed. Connections remain open while there
are any live references, but are now actively closed when all references
are released.
Fixes#67873
Backport of #77295
* Memory efficient xcontent filtering (backport of #77154)
I found myself needing support for something like `filter_path` on
`XContentParser`. It was simple enough to plug it in so I did. Then I
realized that it might offer more memory efficient source filtering
(#25168) so I put together a quick benchmark comparing the source
filtering that we do in `_search`.
Filtering using the parser is about 33% faster than how we filter now
when you select a single field from a 300 byte document:
```
Benchmark (excludes) (includes) (source) Mode Cnt Score Error Units
FetchSourcePhaseBenchmark.filterObjects message short avgt 5 2360.342 ± 4.715 ns/op
FetchSourcePhaseBenchmark.filterXContentOnBuilder message short avgt 5 2010.278 ± 15.042 ns/op
FetchSourcePhaseBenchmark.filterXContentOnParser message short avgt 5 1588.446 ± 18.593 ns/op
```
The top line is the way we filter now. The middle line is adding a
filter to `XContentBuilder` - something we can do right now without any
of my plumbing work. The bottom line is filtering on the parser,
requiring all the new plumbing.
This isn't particularly impresive. 33% *sounds* great! But 700
nanoseconds per document isn't going to cut into anyone's search times.
If you fetch a thousand docuents that's .7 milliseconds of savings.
But we mostly advise folks to use source filtering on fetch when the
source is large and you only want a small part of it. So I tried when
the source is about 4.3kb and you want a single field:
```
Benchmark (excludes) (includes) (source) Mode Cnt Score Error Units
FetchSourcePhaseBenchmark.filterObjects message one_4k_field avgt 5 5957.128 ± 117.402 ns/op
FetchSourcePhaseBenchmark.filterXContentOnBuilder message one_4k_field avgt 5 4999.073 ± 96.003 ns/op
FetchSourcePhaseBenchmark.filterXContentonParser message one_4k_field avgt 5 3261.478 ± 48.879 ns/op
```
That's 45% faster. Put another way, 2.7 microseconds a document. Not
bad!
But have a look at how things come out when you want a single field from
a 4 *megabyte* document:
```
Benchmark (excludes) (includes) (source) Mode Cnt Score Error Units
FetchSourcePhaseBenchmark.filterObjects message one_4m_field avgt 5 8266343.036 ± 176197.077 ns/op
FetchSourcePhaseBenchmark.filterXContentOnBuilder message one_4m_field avgt 5 6227560.013 ± 68306.318 ns/op
FetchSourcePhaseBenchmark.filterXContentonParser message one_4m_field avgt 5 1617153.472 ± 80164.547 ns/op
```
These documents are very large. I've encountered documents like them in
real life, but they've always been the outlier for me. But a 6.5
millisecond per document savings ain't anything to sneeze at.
Take a look at what you get when I turn on gc metrics:
```
FetchSourcePhaseBenchmark.filterObjects message one_4m_field avgt 5 7036097.561 ± 84721.312 ns/op
FetchSourcePhaseBenchmark.filterObjects:·gc.alloc.rate message one_4m_field avgt 5 2166.613 ± 25.975 MB/sec
FetchSourcePhaseBenchmark.filterXContentOnBuilder message one_4m_field avgt 5 6104595.992 ± 55445.508 ns/op
FetchSourcePhaseBenchmark.filterXContentOnBuilder:·gc.alloc.rate message one_4m_field avgt 5 2496.978 ± 22.650 MB/sec
FetchSourcePhaseBenchmark.filterXContentonParser message one_4m_field avgt 5 1614980.846 ± 31716.956 ns/op
FetchSourcePhaseBenchmark.filterXContentonParser:·gc.alloc.rate message one_4m_field avgt 5 1.755 ± 0.035 MB/sec
```
* Fixup benchmark for 7.x
Today `AbstractRefCounted` has a `name` field which is only used to
construct the exception message when calling `incRef()` after it's been
closed. This isn't really necessary, the stack trace will identify the
reference in question and give loads more useful detail besides. It's
also slightly irksome to have to name every single implementation.
This commit drops the name and the constructor parameter, and also
introduces a handy factory method for use when there's no extra state
needed and you just want to run a method or lambda when all references
are released.
* Reformatting to keep Checkstyle after formatting
* Configure spotless everywhere, and disable the tasks if necessary
* Add XContentBuilder helpers, fix test
* Fix copyCurentStructure(MapXContentParser) (#76357)
This stops `MapXContentParser` from throwing an
`UnsupportedOperationException` when passed as an argument to
`XContentBuilder#copyCurrentStructure`. This is mostly useful in tests
where `Map` is a convenient way to talk about structured configuration
but the production APIs need the map to be embedded into a larger blob
of `XContent`.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Fixup
Co-authored-by: Nik Everett <nik9000@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
A MultiCommand requires that a (sub)command is provided for each
execution. Previously, the error handling for MultiCommand would
simply state "ERROR: Missing command" but give no assistance about
what commands were accepted. The user was required to pass "-help" in
order to get the command list.
This commit changes the behaviour so that the list of commands is
printed after the error message.
Backport of: #75993
Today, writing a Writable value to XContent in Base64 format performs
these steps: (1) create a BytesStreamOutput, (2) write Writable to that
output, (3) encode a copy of bytes from that output stream, (4) create a
string from the encoded bytes, (5) write the encoded string to XContent.
These steps allocate/use memory 5 times than writing the encode chars
directly to the output of XContent.
This API would help reduce memory usage when storing a large response
of an async search.
Relates #67594
ParseField is part of the x-content lib, yet it doesn't exist under the
same root package as the rest of the lib. This commit moves the class to
the appropriate package.
relates #73784
When libs/core was created, several classes were moved from server's
o.e.common package, but they were not moved to a new package. Split
packages need to go away long term, so that Elasticsearch can even think
about modularization. This commit moves all the classes under o.e.common
in core to o.e.core.
relates #73784
backport #73909
The recent upgrade of the Azure SDK has caused a few test failures that
have been difficult to debug and do not yet have a fix. In particular, a
change to the netty reactor resolving
(reactor/reactor-netty#1655). We need to wait
for a fix for that issue, so this reverts commit
f454cefc26.
relates #73493
The plugin classloader exists in its own jar file for legacy reasons,
and while it should go away in the future, it currently duplicates the
package name of the rest of the plugin classes. This commit moves the
classloader into its own unique package.
relates #73784
The org.elasticsearch.bootstrap package exists in server with classes
for starting up Elasticsearch. The elasticsearch-core jar has a handful
of classes that were split out from there, namely java version parsing
and jarhell. This commit moves those classes to a new
org.elasticsearch.jdk package so as to not split the server owned
bootstrap package.
relates #73784
This commit upgrades the Azure SDK to 12.11.0 and Jackson to 2.12.2. The
Jackson upgrade must happen at the same time due to Azure depending on
this new version of Jackson.
closes#66555closes#67214
backport #72995
backport #73011
Co-authored-by: Francisco Fernández Castaño <francisco.fernandez.castano@gmail.com>
Co-authored-by: Mark Vieira <portugee@gmail.com>
