We don't enforce Java modularity when executing tests, so we shouldn't
bother highlighting issues in test framework code. This PR adds a new
IntelliJ scope that excludes everything under `:test` and configures the
illegal module dependency inspection to use this new scope.
Closes#86945.
The IntelliJ Checkstyle plugin likes to modify its configuration file
unecessarily, which creates friction for developers because they have
the revert these changes.
Instead, stop versioning the config file and instead update the
`configureIdeCheckstyle` task to copy the required content into place.
This PR represents the initial phase of Modularizing Elasticsearch (with
Java Modules).
This initial phase modularizes the core of the Elasticsearch server
with Java Modules, which is then used to load and configure extension
components atop the server. Only a subset of extension components are
modularized at this stage (other components come in a later phase).
Components are loaded dynamically at runtime with custom class loaders
(same as is currently done). Components with a module-info.class are
defined to a module layer.
This architecture is somewhat akin to the Modular JDK, where
applications run on the classpath. In the analogy, the Elasticsearch
server modules are the platform (thus are always resolved and present),
while components without a module-info.class are non-modular code
running atop the Elasticsearch server modules. The extension components
cannot access types from non-exported packages of the server modules, in
the same way that classpath applications cannot access types from
non-exported packages of modules from the JDK. Broadly, the core
Elasticseach java modules simply "wrap" the existing packages and export
them. There are opportunites to export less, which is best done in more
narrowly focused follow-up PRs.
The Elasticsearch distribution startup scripts are updated to put jars
on the module path (the class path is empty), so the distribution will
run the core of the server as java modules. A number of key components
have been retrofitted with module-info.java's too, and the remaining
components can follow later. Unit and functional tests run as
non-modular (since they commonly require package-private access), while
higher-level integration tests, that run the distribution, run as
modular.
Co-authored-by: Chris Hegarty <christopher.hegarty@elastic.co>
Co-authored-by: Ryan Ernst <ryan@iernst.net>
Co-authored-by: Rene Groeschke <rene@elastic.co>
Closes#53812.
The CheckStyle-IDEA and EcliseCodeFormatter IDEA plugins are to all
intents and purposes required when using IntelliJ to work on
Elasticsearch, so add configuration that marks then as required.
Utilize the separate artifacts for the native ML code and any platform
native dependencies. This improves dependency cachability since these
native dependencies rarely change with updates to native code and
amount to about 90% of the total artifact size.
Checkstyle 9.3 has been released, and contains a fix for the memory
problems we saw in previous versions. Upgrade to this version.
Note that the IDEA config has also been updated, but the plugin
doesn't yet have 9.3 as an option according to the release notes.
This shouldn't be a problem.
Our spotless configuration wasn't being applied to `build-tools`
and `build-tools-internal`. Move the Spotless configuration to a
Java plugin in `build-conventions`, and apply it everywhere.
This resulted in a lot more Java files being subject to formatting,
so I added more exclusions to the list.
Also remove the `paddedCell` stuff, we've never needed it.
Adds a new service for trained model allocation to nodes.
Initially, this only supports PyTorch models and simply allocates
to nodes with the ML roles.
Design is fairly simple:
- A master node service runs allowing for new allocations to be created/updated/deleted from cluster state
- A node service runs listening to updates referencing the local node + any models it may have allocated and updates accordingly.
This type of service sort of splits the difference between the logic of shard allocation and persistent tasks. Neither really fully addressed the need here.
Change the formatter config to sort / order imports, and reformat the
codebase. We already had a config file for Eclipse users, so Spotless now
uses that.
The "Eclipse Code Formatter" plugin ought to be able to use this file as
well for import ordering, but in my experiments the results were poor.
Instead, use IntelliJ's `.editorconfig` support to configure import
ordering.
I've also added a config file for the formatter plugin.
Other changes:
* I've quietly enabled the `toggleOnOff` option for Spotless. It was
already possible to disable formatting for sections using the markers
for docs snippets, so enabling this option just accepts this reality
and makes it possible via `formatter:off` and `formatter:on` without
the restrictions around line length. It should still only be used as
a very last resort and with good reason.
* I've removed mention of the `paddedCell` option from the contributing
guide, since I haven't had to use that option for a very long time. I
moved the docs to the spotless config.
When remote run configuration is executed in IntelliJ Community edition, it
automatically adds the RunnedSettings section. This section doesn't seem to have
any negative effect on the Ultimate Edition.
This commit makes a number of improvements when importing the
Elasticsearch project into IntelliJ IDEA. Specifically:
- Contributing documentation has been updated to reflect that the
'idea' task should no long be used and Gradle project import is
instead the officially supported way of setting up the project.
- Attempts to run the 'idea' task will result in a failure with a
message directing folks to our CONTRIBUTING.md document.
- The project JDK is explicit set rather that using whatever JAVA_HOME
is.
- Gradle build operation delegation is disabled, and test execution is
configured to 'choose per test'.
- Gradle is configured to inherit the project JDK.
- Some code style conventions are automatically configured.
- File encoding is explicitly set to UTF-8.
- Parallel module compilation is enabled and deprecated feature
warnings are disabled.
- A remote debug run configuration using listen mode is created.
- JUnit runner is configured with required system properties.
- License headers are configured such that Apache 2 is the default
notice added to all source files with exception of source in /x-pack
which will use the Elastic license.
