This updates the gradle wrapper to 8.12
We addressed deprecation warnings due to the update that includes:
- Fix change in TestOutputEvent api
- Fix deprecation in groovy syntax
- Use latest ospackage plugin containing our fix
- Remove project usages at execution time
- Fix deprecated project references in repository-old-versions
replace deprecated gradle api
fix permission api usage in debian and rpm package creation
remove deprecated usage of #ProjectDependency..getDependencyProject()
improves gradle configuration cache reading in our weekly benchmark by almost 30%
Static fields dont do well in Gradle with configuration cache enabled.
- Use buildParams extension in build scripts
- Keep BuildParams.ci for now for easy serverless migration
- Tweak testing doc
This commit removes `com.google.crypto.tink` which is transitive and optional dependency of `oauth2-oidc-sdk` and `nimbus-jose-jwt`. We don't seem to be using any functionality that requires `tink` and thus `protobuf-java`. Removing them feels safer than having to maintain misaligned versions.
The most relevant ES changes that upgrading to Lucene 10 requires are:
- use the appropriate IOContext
- Scorer / ScorerSupplier breaking changes
- Regex automaton are no longer determinized by default
- minimize moved to test classes
- introduce Elasticsearch900Codec
- adjust slicing code according to the added support for intra-segment concurrency
- disable intra-segment concurrency in tests
- adjust accessor methods for many Lucene classes that became a record
- adapt to breaking changes in the analysis area
Co-authored-by: Christoph Büscher <christophbuescher@posteo.de>
Co-authored-by: Mayya Sharipova <mayya.sharipova@elastic.co>
Co-authored-by: ChrisHegarty <chegar999@gmail.com>
Co-authored-by: Brian Seeders <brian.seeders@elastic.co>
Co-authored-by: Armin Braun <me@obrown.io>
Co-authored-by: Panagiotis Bailis <pmpailis@gmail.com>
Co-authored-by: Benjamin Trent <4357155+benwtrent@users.noreply.github.com>
- Replaced AWS 1.12.740 with 2.28.13
- Removed `aws-java-sdk*` and its transitive dependencies.
- Added `awssdk:bedrockruntime` as an `implementations`, all transitive
dependencies are added as `api` matching their marked `Compile` in
maven.
- Added `awssdk:netty-nio-client` as our client implementation, since
our v1 integration is using the respective Async client.
- Added netty packages as `runtimeOnly` since they are only used during
runtime.
- Replaced AWS's use of SLF4J-1.7 with our declaration of SLF4J-2.x,
since SLF4J includes backwards-compatible bindings.
- Migrated all references from the v1 package (`com.amazonaws`) to the
v2 package (`software.amazon.awssdk`).
Notable changes in the SDK:
- *Result objects are renamed to *Response objects.
- Objects are now immutable and require Builders to set fields.
- Getters no longer have the `get*` prefix, e.g. `getModelId()` is now
`modelId()`.
- `Future` has been replaced with `CompletableFuture`.
- There is no longer a need to invoke the `IdleConnectionReaper`, this
is now done when the client is closed.
- Builders have a consumer mutation pattern for modifying many fields at
once.
Security changes:
- The underlying Builder objects always check to see if the
`.aws/credentials` and `.aws/config` files exist, even if they are not
used, so our `plugin-security.policy` now allows reading these files.
- The Builder always checks for the `http.proxyHost` property before
defaulting to the hardcoded Bedrock URL.
Resolve#110590
This commit bumps the Byte Buddy version to 1.14.12. This is needed to support class file versions generated by Java 21.
We already use this version of Byte Buddy elsewhere, so there is not need to update the gradle verification metadata.
This change is in preparation for an eventual bump of Elasticsearch to a minimum of JDK 21, in ES 9.0.
`azure-core-1.51.0` includes fixes for a handful of bugs we reported.
This commit upgrades the `azure-storage-blob` and `azure-identity`
dependencies to pick up this new fixed version of `azure-core`.
This commit bumps the asm version to 9.7 in test logger usage, which contains support for newer Java class file versions.
This change is in preparation for an eventual bump of Elasticsearch to a minimum of JDK 21, in ES 9.0
This commit updates the Nimbus OAuth2 OIDC SDK and the associated Nimbus JOSE+JWT, however a few odd choices had to be made in the process.
First, we update to versions which are old at time of merge. This is because versions of Nimbus JOSE+JWT 9.38 and after through time of writing [contain a bug](https://bitbucket.org/connect2id/nimbus-jose-jwt/issues/550/java-module-doesnt-work-properly-with) in which the shaded gson class files included in the library are not properly loaded by our module loading code (and possibly in general? the root cause of the bug is unclear at time of writing but it does not appear to be present in all uses of this library). This requires us to use an older version of Nimbus OAuth2 OIDC SDK as well.
Second, the aforementioned shaded gson uses reflection internally, and is used in unpredictable places in these libraries (e.g. constructors). This is extremely unfriendly to our usage of the security manager. In order to make the scope of permission grants as narrow as possible, we shadow nimbus-jose-jwt in order to insert `AccessController.doPrivileged` calls at the appropriate points, given the usage of gson is relatively contained.
This approach was chosen over other approaches given 1) the relative simplicity given the implementation of the library, and 2) the complexity involved in safely using the library any other way - as one example, gson is used frequently in `toString()` methods, which are frequently called implicitly, especially in combination with logging which may mask security manager exceptions from being surfaced in tests. All of the code we intercept should be re-evaluated when this library is next upgraded.
Co-authored-by: Jake Landis <jake.landis@elastic.co>
With this commit, if no key or SAS token is supplied for an Azure
repository then Elasticsearch will use the `DefaultAzureCredential`
chain defined in the Azure SDK, which will obtain credentials from the
instance metadata service when running on an Azure VM.
* Initial commit; setup Gradle; start service
* initial commit
* minor cleanups, builds green; needs tests
* bug fixes; tested working embeddings & completion
* use custom json builder for embeddings request
* Ensure auto-close; fix forbidden API
* start of adding unit tests; abstraction layers
* adding additional tests; cleanups
* add requests unit tests
* all tests created
* fix cohere embeddings response
* fix cohere embeddings response
* fix lint
* better test coverage for secrets; inference client
* update thread-safe syncs; make dims/tokens + int
* add tests for dims and max tokens positive integer
* use requireNonNull;override settings type;cleanups
* use r/w lock for client cache
* remove client reference counting
* update locking in cache; client errors; noop doc
* remove extra block in internalGetOrCreateClient
* remove duplicate dependencies; cleanup
* add fxn to get default embeddings similarity
* use async calls to Amazon Bedrock; cleanups
* use Clock in cache; simplify locking; cleanups
* cleanups around executor; remove some instanceof
* cleanups; use EmbeddingRequestChunker
* move max chunk size to constants
* oof - swapped transport vers w/ master node req
* use XContent instead of Jackson JsonFactory
* remove gradle versions; do not allow dimensions
Initial support for Apache Arrow's streaming format as a response for ES|QL. It triggers based on the Accept header or the format request parameter.
Arrow has implementations in every mainstream language and is a backend of the Python Pandas library, which is extremely popular among data scientists and data analysts. Arrow's streaming format has also become the de facto standard for dataframe interchange. It is an efficient binary format that allows zero-cost deserialization by adding data access wrappers on top of memory buffers received from the network.
This PR builds on the experiment made by @nik9000 in PR #104877
Features/limitations:
- all ES|QL data types are supported
- multi-valued fields are not supported
- fields of type _source are output as JSON text in a varchar array. In a future iteration we may want to offer the choice of the more efficient CBOR and SMILE formats.
Technical details:
Arrow comes with its own memory management to handle vectors with direct memory, reference counting, etc. We don't want to use this as it conflicts with Elasticsearch's own memory management.
We therefore use the Arrow library only for the metadata objects describing the dataframe schema and the structure of the streaming format. The Arrow vector data is produced directly from ES|QL blocks.
---------
Co-authored-by: Nik Everett <nik9000@gmail.com>
* Update Gradle gradleEnterprise plugin (develocity now)
* Fix imports in build scan script
* Fix build scan api usage
* Dependency cleanup and API fix
* Fix API update for BuildResult in Build scans
* Fix buildkite buildscan annotations based on gradle failures
Fix incompatibility with 8.8 and our internal api usages
- Update ospackage to a version that contains a fix we provided
- Tweak build logic to avoid deprecation warnings
- Use newer permission api
- Use custom shadowplugin
- Rework ElasticsearchDistribution dependencies resolution
- Update Gradle wrapper to 8.8
