* Remove @UpdateForV9 annotation in RuntimeVersionFeature
Re ES-10338 the Core/Infra team agreed that this is not blocked on security manager removal. We should go ahead and remove these 2 instances of the @UpdateForV9 annotation, the code will be cleaned up soon with the removal of Security Manager.
* Remove @UpdateForV9 annotation from SystemJvmOptions
re ES-10338 the Core/Infra team agreed this is not blocked on security manager removal. We should go ahead and remove this instance of the @UpdateForV9 annotation.
Bit of a random find. Looks like we're effectively spending something
like ~0.5% of the total CPU time on resolving virtual calls for these
methods. Inlining `IOUtils` reduces their size quite a bit, removes
dead-code for rethrowing `IOException` that is never actually thrown
and hopefully as a result improve compilation a little here.
Currently the SLM health indicator in health report turns YELLOW when snapshots fail for a number of times. However, the SLM health indicator stays GREEN if snapshot is not completed (no success or failure) for a long time. This change adds a new optional setting unhealthy_if_no_snapshot_within to SLM policy, that sets a time threshold. If the SLM policy has not had a successful snapshot for longer than the threshold, the SLM health indicator will turn YELLOW.
This commit adjusts the common lib of entitlement tools to use
elasticsearch.build so that it gets java version configuration
automatically. Additionally the mrjar plugin is removed from the core
lib since it is not used there.
A ProjectResolver should be able to fetch the ProjectId without the need
of a cluster state. This PR changes the method to take no argument and
fixes cascading changes. It also adds a separate ProjectIdResolver
interface to host the new no-arg getProjectId method. The existing
ProjectResolver interface extends the new interface.
The major impact of the change is for tests and test helpers. It has
been made more obvious that most tests rely on the default project ID.
Previously this was hidden by the no arg `singleProjectOnly()` that just
pops the only project from cluster state. This only project is the
default project anyway in the current tests. The change just made this
fact explicit which I think is not a bad thing. We can definitely tidy
up the tests and test helpers once more pieces fall into places. Happy
to take suggestions.
* Refactor: treat "maybe" JVM options uniformly
* WIP
* Get entitlement running with bridge all the way through, with qualified
exports
* Cosmetic changes to SystemJvmOptions
* Disable entitlements by default
* Bridge module comments
* Fixup forbidden APIs
* spotless
* Rename EntitlementChecker
* Fixup InstrumenterTests
* exclude recursive dep
* Fix some compliance stuff
* Rename asm-provider
* Stop using bridge in InstrumenterTests
* Generalize readme for asm-provider
* InstrumenterTests doesn't need EntitlementCheckerHandle
* Better javadoc
* Call parseBoolean
* Add entitlement to internal module list
* Docs as requested by Lorenzo
* Changes from Jack
* Rename ElasticsearchEntitlementChecker
* Remove logging javadoc
* exportInitializationToAgent should reference EntitlementInitialization, not EntitlementBootstrap.
They're currently in the same module, but if that ever changes, this code would have become wrong.
* Some suggestions from Mark
---------
Co-authored-by: Ryan Ernst <ryan@iernst.net>
The libs projects are configured to all begin with `elasticsearch-`.
While this is desireable for the artifacts to contain this consistent
prefix, it means the project names don't match up with their
directories. Additionally, it creates complexities for subproject naming
that must be manually adjusted.
This commit adjusts the project names for those under libs to be their
directory names. The resulting artifacts for these libs are kept the
same, all beginning with `elasticsearch-`.
This annotation (which will be removed in the future) marks places in
the source code that we need to revisit as part of the multi-project
initiative.
It can be used to mark
* code that should be removed,
* code where we have made decisions that we'd like to revisit
* places that are known to break when operating on multiple projects
* anything else that needs to be looked at before the code can be
considered production-ready
* Entitlements for System.exit
* Respond to Simon's comments
* Rename trampoline -> bridge
* Require exactly one bridge jar
* Use Type helpers to generate descriptor strings
* Various cleanup from PR comments
* Remove null "receiver" for static methods
* Use List<Type> instead of voidDescriptor
* Clarifying comment
* Whoops, getMethod
* SuppressForbidden System.exit
* Spotless
* Use embedded provider plugin to keep ASM off classpath
* Oops... forgot the punchline
* Move ASM license to impl
* Use ProviderLocator and simplify bridgeJar logic
* Avoid eager resolution of configurations during task configuration
* Remove compile-time dependency agent->bridge
---------
Co-authored-by: Mark Vieira <portugee@gmail.com>
In order to better track the work needed to prepare this branch for the
major version upgrade, this commit adds a mandatory `owner` field to all
`UpdateForV9` (and `UpdateForV10`) annotations.
The overridden `RefCounted` methods on `AbstractRefCounted` are all
`final`, but `mustIncRef` uses the default implementation and can still
be overridden. For the avoidance of doubt, this commit marks
`mustIncRef` as `final`.
This commit bumps the REST API version from 8 to 9. This effectively removes all support for REST API
compatibility with version 7 (though earlier commits already chipped away at some v7 support).
This also enables REST API compatibility support for version 8, providing support for v8 compatibility headers,
i.e. "application/vnd.elasticsearch+json;compatible-with=8" and no-op support (no errors) to accept v9
compatibility headers i.e. "application/vnd.elasticsearch+json;compatible-with=9".
see additional context in the GH PR #113151
No need to precompute the toString for `ActionListener` and
`Releasable`, that's quite expensive at times. Also string concat is way
faster than formating these days, so use that in the transport channels.
Lastly, short-circuit some obvious spots in network address
serialization and remove code that duplicates the JDK (remove the IPV4
specific forbidden API because it makes no sense, but still needed to
disable the check to make the build green because of the exclude on the
parent class).
Follow up to #110552, add utility for starting tasks at the same time
via a barrier as discussed there. Also, make use of the new tooling in a
couple more spots to save LoC and thread creation.
This creates the infrastructure to allow optimizing the `lookup` method
when applied to `Vector`s and then implements that optimization for
constant vectors. Constant vectors now take one of six paths:
1. An empty positions `Block` yields an empty result set.
2. If `positions` is a `Block`, perform the un-optimized lookup.
3. If the `min` of the `positions` *Vector* is less that 0 then throw an
exception.
4. If the `min` of the positions Vector is greater than the number of
positions in the lookup block then return a single
`ConstantNullBlock` because you are looking up outside the range.
5. If the `max` of the positions Vector is less than the number of
positions in the lookup block then return a `Constant$Type$Block`
with the same value as the lookup block. This is a lookup that's
entirely within range.
6. Otherwise return the unoptimized lookup.
This is *fairly* simple but demonstrates how we can plug in more complex
optimizations later.
This adds a /_capabilities rest endpoint for checking the capabilities of a cluster - what endpoints, parameters, and endpoint capabilities the cluster supports
The values `30s` and `1m` are used as defaults in various places in ES,
there's no need to create a new `TimeValue` instance each time they
appear. Moreover we already have constants for `0` and `-1`, but we
don't use these constants when reading the values off the wire.
This commit adds constants for `30s` and `1m` and adjusts the
deserialization code to avoid unnecessary allocation for common
`TimeValue` instances.
Relates #107984
Adds a `lookup` method to `BlockHash` which finds keys that were already
in the hash without modifying it and returns the "ordinal" that the
`BlockHash` produced when that key had been called with `add`.
For multi-column keys this can change the number of values pretty
drastically. You get a combinatorial explosion of values. So if you have
three columns with 2 values each the most values you can get is 2*2*2=8.
If you have five columns with ten values each you can have 100,000
values in a single position! That's too many.
Let's do an example! This one has a two row block containing three
colunms. One row has two values in each column so it could produce at
most 8 values. In this case one of the values is missing from the hash,
so it only produces 7.
Block:
| a | b | c |
| ----:| ----:| ----:|
| 1 | 4 | 6 |
| 1, 2 | 3, 4 | 5, 6 |
BlockHash contents:
| a | b | c |
| -:| -:| -:|
| 1 | 3 | 5 |
| 1 | 3 | 6 |
| 1 | 4 | 5 |
| 1 | 4 | 6 |
| 2 | 3 | 5 |
| 2 | 3 | 6 |
| 2 | 4 | 6 |
Results:
| ord |
| -------------------:|
| 3 |
| 0, 1, 2, 3, 4, 5, 6 |
The `add` method has a fairly fool-proof mechanism to work around this,
it calls it's consumers with a callback that can split positions into
multiple calls. It calls the callback in batches of like 16,000
positions at a time. And aggs uses the callback. So you can aggregate
over five colunms with ten values each. It's slow, but the callbacks
let us get through it.
Unlike `add`, `lookup` can't use a callback. We're going to need it to
return `Iterator` of `IntBlock`s containing ordinals. That's just how
we're going to use it. That'd be ok, but we can't split a single
position across multiple `Block`s. That's just not how `Block` works.
So, instead, we fail the query if we produce more than 100,000 entries
in a single position. We'd like to stop collecting and emit a warning,
but that's a problem for another change. That's a single 400kb array
which is quite big.
Anyway! If we're not bumping into massive rows we emit `IntBlocks`
targeting a particular size in memory. Likely we'll also want to plug in
a target number of rows as well, but for now this'll do.
Today various test utility methods construct a random string that can be
parsed as a `TimeValue`, but in practice almost everywhere we just parse
it straight away. With this change we have the utility methods return a
`TimeValue` directly.
Just a suggetion. I think this would save us a bit of memory here and
there. We have loads of places where the always true lambdas are used
with `Predicate.or/and`. Found this initially when looking into field
caps performance where we used to heavily compose these but many spots
in security and index name resolution gain from these predicates.
The better toString also helps in some cases at least when debugging.
Introducing a plain version of `AbstractRefCounted` as a compromise.
This saves a bunch of allocations and a circular reference to the object
holding the ref counted instance, making smaller SearchHit instances
etc. cheaper. We could get an even more direct solution here by making
these extend `AbstractRefCounted` but that would lose us the ability to
leak-track in tests, so doing it this way (same way Netty does it on
their end) as a compromise.
This is mainly added as a prerequisite to slicing doc sources out of
bulk index requests. The few usages for it added in this PR have limited
performance impact but demonstrate correct functioning of the
implementations.
Co-authored-by: David Turner <david.turner@elastic.co>
Elasticsearch requires access to some native functions. Historically
this has been achieved with the JNA library. However, JNA is a
complicated, magical library, and has caused various problems booting
Elasticsearch over the years. The new Java Foreign Function and Memory
API allows access to call native functions directly from Java. It also
has the advantage of tight integration with hotspot which can improve
performance of these functions (though performance of Elasticsearch's
native calls has never been much of an issue since they are mostly at
boot time).
This commit adds a new native lib that is internal to Elasticsearch. It
is built to use the foreign function api starting with Java 21, and
continue using JNA with Java versions below that.
Only one function, checking whether Elasticsearch is running as root, is
migrated. Future changes will migrate other native functions.
Qualfied exports in the boot layer only work when they are to other boot
modules. Yet Elasticsearch has dynamically loaded modules as in plugins.
For this purpose we have ModuleQualifiedExportsService. This commit
moves loading of ModuleQualfiedExportService instances in the boot layer
into core so that it can be reused by ProviderLocator when a qualified
export applies to an embedded module.
It's less code and it actually inlines (avoiding virtual calls in most
cases) to just do the null check here instead of delegating to IOUtils
and then catching the impossible IOException. Also, no need to use
`Releaseables` in 2 spots where try-with-resources works as well and
needs less code.
Noticed this when I saw that we had a lot of strange CPU overhead in
this call in some hot loops like translog writes.
This commit adds an elasticsearch gradle plugin which sets up a project
to build an MRJAR. By applying the plugin, the src dir is checked for
any directories matching mainXX where XX is the java version number.
A source set is automatically setup, an appropriate compiler tied to
that source set, and it's output placed in the correct part of the final
jar. Additionally, the sourceset allows use of preview features in that
verison of Java, and the preview bits are stripped from the resulting
class files.
This is needed for the search response pooling work. Also, the one usage
in `ReleasableBytesReference` actually makes outright sense. We
shouldn't be ref-counting on a global constant, that just needlessly
introduces contention that isn't entirely obvious. This change required
a couple tests to be adjusted where we were checking release mechanics
on noop instances.
Today we `decRef()` the per-node responses just after adding them to the
`responses` collection, but in fact we should keep them alive until
we've constructed the final response. This commit does that.
In several places we acquire a ref to a resource that we are certain is
not closed, so this commit adds a utility for asserting this to be the
case. This also helps a little with mocks since boolean methods like
`tryIncRef()` return `false` on mock objects by default, but void
methods like `mustIncRef()` default to being a no-op.
