[[release-notes-8.9.0]] == {es} version 8.9.0 Also see <>. [[known-issues-8.9.0]] [float] === Known issues * Question Answering fails on long input text. If the context supplied to the task is longer than the model's max_sequence_length and truncate is set to none then inference fails with the message `question answering result has invalid dimension`. (issue: {es-issue}97917[#97917]) include::8.7.1.asciidoc[tag=no-preventive-gc-issue] [[breaking-8.9.0]] [float] === Breaking changes Aggregations:: * Switch TDigestState to use `HybridDigest` by default {es-pull}96904[#96904] [[bug-8.9.0]] [float] === Bug fixes Allocation:: * Attempt to fix delay allocation {es-pull}95921[#95921] * Fix NPE in Desired Balance API {es-pull}97775[#97775] * Fix autoexpand during node replace {es-pull}96281[#96281] Authorization:: * Resolving wildcard application names without prefix query {es-pull}96479[#96479] (issue: {es-issue}96465[#96465]) CRUD:: * Fix `retry_on_conflict` parameter in update API to not retry indefinitely {es-pull}96262[#96262] * Handle failure in `TransportUpdateAction#handleUpdateFailureWithRetry` {es-pull}97290[#97290] (issue: {es-issue}97286[#97286]) Cluster Coordination:: * Avoid `getStateForMasterService` where possible {es-pull}97304[#97304] * Become candidate on publication failure {es-pull}96490[#96490] (issue: {es-issue}96273[#96273]) * Fix cluster settings update task acknowledgment {es-pull}97111[#97111] Data streams:: * Accept timestamp as object at root level {es-pull}97401[#97401] Geo:: * Fix bug when creating empty `geo_lines` {es-pull}97509[#97509] (issue: {es-issue}97311[#97311]) * Fix time-series geo_line to include reduce phase in MergedGeoLines {es-pull}96953[#96953] (issue: {es-issue}96983[#96983]) * Support for Byte and Short as vector tiles features {es-pull}97619[#97619] (issue: {es-issue}97612[#97612]) ILM+SLM:: * Limit the details field length we store for each SLM invocation {es-pull}97038[#97038] (issue: {es-issue}96918[#96918]) Infra/CLI:: * Initialise ES logging in CLI {es-pull}97353[#97353] (issue: {es-issue}97350[#97350]) Infra/Core:: * Capture max processors in static init {es-pull}97119[#97119] (issue: {es-issue}97088[#97088]) * Interpret microseconds cpu stats from cgroups2 properly as nanos {es-pull}96924[#96924] (issue: {es-issue}96089[#96089]) Infra/Logging:: * Add slf4j-nop in order to prevent startup warnings {es-pull}95459[#95459] Infra/REST API:: * Fix tchar pattern in `RestRequest` {es-pull}96406[#96406] Infra/Scripting:: * Fix Painless method lookup over unknown super interfaces {es-pull}97062[#97062] (issue: {es-issue}97022[#97022]) Infra/Settings:: * Enable validation for `versionSettings` {es-pull}95874[#95874] (issue: {es-issue}95873[#95873]) Ingest Node:: * Fixing `DateProcessor` when the format is `epoch_millis` {es-pull}95996[#95996] * Fixing `GeoIpDownloaderStatsAction$NodeResponse` serialization by defensively copying inputs {es-pull}96777[#96777] (issue: {es-issue}96438[#96438]) * Trim field references in reroute processor {es-pull}96941[#96941] (issue: {es-issue}96939[#96939]) Machine Learning:: * Catch exceptions thrown during inference and report as errors {ml-pull}2542[#2542] * Fix `WordPiece` tokenization where stripping accents results in an empty string {es-pull}97354[#97354] * Improve model downloader robustness {es-pull}97274[#97274] * Prevent high memory usage by evaluating batch inference singularly {ml-pull}2538[#2538] Mapping:: * Avoid stack overflow while parsing mapping {es-pull}95705[#95705] (issue: {es-issue}52098[#52098]) * Fix mapping parsing logic to determine synthetic source is active {es-pull}97355[#97355] (issue: {es-issue}97320[#97320]) Ranking:: * Fix `sub_searches` serialization bug {es-pull}97587[#97587] Recovery:: * Promptly fail recovery from snapshot {es-pull}96421[#96421] (issue: {es-issue}95525[#95525]) Search:: * Prevent instantiation of `top_metrics` when sub-aggregations are present {es-pull}96180[#96180] (issue: {es-issue}95663[#95663]) * Set new providers before building `FetchSubPhaseProcessors` {es-pull}97460[#97460] (issue: {es-issue}96284[#96284]) Snapshot/Restore:: * Fix blob cache races/assertion errors {es-pull}96458[#96458] * Fix reused/recovered bytes for files that are only partially recovered from cache {es-pull}95987[#95987] (issues: {es-issue}95970[#95970], {es-issue}95994[#95994]) * Fix reused/recovered bytes for files that are recovered from cache {es-pull}97278[#97278] (issue: {es-issue}95994[#95994]) * Refactor `RestoreClusterStateListener` to use `ClusterStateObserver` {es-pull}96662[#96662] (issue: {es-issue}96425[#96425]) TSDB:: * Error message for misconfigured TSDB index {es-pull}96956[#96956] (issue: {es-issue}96445[#96445]) * Min score for time series {es-pull}96878[#96878] Task Management:: * Improve cancellability in `TransportTasksAction` {es-pull}96279[#96279] Transform:: * Improve reporting status of the transform that is about to finish {es-pull}95672[#95672] [[enhancement-8.9.0]] [float] === Enhancements Aggregations:: * Add cluster setting to `SearchExecutionContext` to configure `TDigestExecutionHint` {es-pull}96943[#96943] * Add support for dynamic pruning to cardinality aggregations on low-cardinality keyword fields {es-pull}92060[#92060] * Make TDigestState configurable {es-pull}96794[#96794] * Skip `SortingDigest` when merging a large digest in `HybridDigest` {es-pull}97099[#97099] * Support value retrieval in `top_hits` {es-pull}95828[#95828] Allocation:: * Take into account `expectedShardSize` when initializing shard in simulation {es-pull}95734[#95734] Analysis:: * Create `.synonyms` system index {es-pull}95548[#95548] Application:: * Add template parameters to Search Applications {es-pull}95674[#95674] * Chunk profiling stacktrace response {es-pull}96340[#96340] * [Profiling] Add status API {es-pull}96272[#96272] * [Profiling] Allow to upgrade managed ILM policy {es-pull}96550[#96550] * [Profiling] Introduce ILM for K/V indices {es-pull}96268[#96268] * [Profiling] Require POST to retrieve stacktraces {es-pull}96790[#96790] * [Profiling] Tweak default ILM policy {es-pull}96516[#96516] * [Search Applications] Support arrays in stored mustache templates {es-pull}96197[#96197] Authentication:: * Header validator with Security {es-pull}95112[#95112] Authorization:: * Add Search ALC filter index prefix to the enterprise search user {es-pull}96885[#96885] * Ensure checking application privileges work with nested-limited roles {es-pull}96970[#96970] Autoscaling:: * Add shard explain info to `ReactiveReason` about unassigned shards {es-pull}88590[#88590] (issue: {es-issue}85243[#85243]) DLM:: * Add auto force merge functionality to DLM {es-pull}95204[#95204] * Adding `data_lifecycle` to the _xpack/usage API {es-pull}96177[#96177] * Adding `manage_data_stream_lifecycle` index privilege and expanding `view_index_metadata` for access to data stream lifecycle APIs {es-pull}95512[#95512] * Allow for the data lifecycle and the retention to be explicitly nullified {es-pull}95979[#95979] Data streams:: * Add support for `logs@custom` component template for `logs-*-* data streams {es-pull}95481[#95481] (issue: {es-issue}95469[#95469]) * Adding ECS dynamic mappings component and applying it to logs data streams by default {es-pull}96171[#96171] (issue: {es-issue}95538[#95538]) * Adjust ECS dynamic templates to support `subobjects: false` {es-pull}96712[#96712] * Automatically parse log events in logs data streams, if their `message` field contains JSON content {es-pull}96083[#96083] (issue: {es-issue}95522[#95522]) * Change default of `ignore_malformed` to `true` in `logs-*-*` data streams {es-pull}95329[#95329] (issue: {es-issue}95224[#95224]) * Set `@timestamp` for documents in logs data streams if missing and add support for custom pipeline {es-pull}95971[#95971] (issues: {es-issue}95537[#95537], {es-issue}95551[#95551]) * Update data streams implicit timestamp `ignore_malformed` settings {es-pull}96051[#96051] Engine:: * Cache modification time of translog writer file {es-pull}95107[#95107] * Trigger refresh when shard becomes search active {es-pull}96321[#96321] (issue: {es-issue}95544[#95544]) Geo:: * Add brute force approach to `GeoHashGridTiler` {es-pull}96863[#96863] * Asset tracking - geo_line in time-series aggregations {es-pull}94954[#94954] ILM+SLM:: * Chunk the GET _ilm/policy response {es-pull}97251[#97251] (issue: {es-issue}96569[#96569]) * Move get lifecycle API to Management thread pool and make cancellable {es-pull}97248[#97248] (issue: {es-issue}96568[#96568]) * Reduce WaitForNoFollowersStep requests indices shard stats {es-pull}94510[#94510] Indices APIs:: * Bootstrap profiling indices at startup {es-pull}95666[#95666] Infra/Node Lifecycle:: * SIGTERM node shutdown type {es-pull}95430[#95430] Ingest Node:: * Add mappings for enrich fields {es-pull}96056[#96056] * Ingest: expose reroute inquiry/reset via Elastic-internal API bridge {es-pull}96958[#96958] Machine Learning:: * Improved compliance with memory limitations {ml-pull}2469[#2469] * Improve detection of calendar cyclic components with long bucket lengths {ml-pull}2493[#2493] * Improve detection of time shifts, for example for daylight saving {ml-pull}2479[#2479] Mapping:: * Allow unsigned long field to use decay functions {es-pull}96394[#96394] (issue: {es-issue}89603[#89603]) Ranking:: * Add multiple queries for ranking to the search endpoint {es-pull}96224[#96224] Recovery:: * Implement `StartRecoveryRequest#getDescription` {es-pull}95731[#95731] Search:: * Add search shards endpoint {es-pull}94534[#94534] * Don't generate stacktrace in `EarlyTerminationException` and `TimeExceededException` {es-pull}95910[#95910] * Feature/speed up binary vector decoding {es-pull}96716[#96716] * Improve brute force vector search speed by using Lucene functions {es-pull}96617[#96617] * Include search idle info to shard stats {es-pull}95740[#95740] (issue: {es-issue}95727[#95727]) * Integrate CCS with new `search_shards` API {es-pull}95894[#95894] (issue: {es-issue}93730[#93730]) * Introduce a filtered collector manager {es-pull}96824[#96824] * Introduce minimum score collector manager {es-pull}96834[#96834] * Skip shards when querying constant keyword fields {es-pull}96161[#96161] (issue: {es-issue}95541[#95541]) * Support CCS minimize round trips in async search {es-pull}96012[#96012] * Support for patter_replace filter in keyword normalizer {es-pull}96588[#96588] * Support null_value for rank_feature field type {es-pull}95811[#95811] Security:: * Add "_storage" internal user {es-pull}95694[#95694] Snapshot/Restore:: * Reduce overhead in blob cache service get {es-pull}96399[#96399] Stats:: * Add `ingest` information to the cluster info endpoint {es-pull}96328[#96328] (issue: {es-issue}95392[#95392]) * Add `script` information to the cluster info endpoint {es-pull}96613[#96613] (issue: {es-issue}95394[#95394]) * Add `thread_pool` information to the cluster info endpoint {es-pull}96407[#96407] (issue: {es-issue}95393[#95393]) TSDB:: * Feature: include unit support for time series rate aggregation {es-pull}96605[#96605] (issue: {es-issue}94630[#94630]) Vector Search:: * Leverage SIMD hardware instructions in Vector Search {es-pull}96453[#96453] (issue: {es-issue}96370[#96370]) [[feature-8.9.0]] [float] === New features Application:: * Enable analytics geoip in behavioral analytics {es-pull}96624[#96624] Authorization:: * Support restricting access of API keys to only certain workflows {es-pull}96744[#96744] Data streams:: * Adding ability to auto-install ingest pipelines and refer to them from index templates {es-pull}95782[#95782] Geo:: * Geometry simplifier {es-pull}94859[#94859] ILM+SLM:: * Enhance ILM Health Indicator {es-pull}96092[#96092] Infra/Node Lifecycle:: * Gracefully shutdown elasticsearch {es-pull}96363[#96363] Infra/Plugins:: * [Fleet] Add `.fleet-secrets` system index {es-pull}95625[#95625] (issue: {es-issue}95143[#95143]) Machine Learning:: * Add support for `xlm_roberta` tokenized models {es-pull}94089[#94089] * Removes the technical preview admonition from query_vector_builder docs {es-pull}96735[#96735] Snapshot/Restore:: * Add repo throttle metrics to node stats api response {es-pull}96678[#96678] (issue: {es-issue}89385[#89385]) Stats:: * New HTTP info endpoint {es-pull}96198[#96198] (issue: {es-issue}95391[#95391]) [[upgrade-8.9.0]] [float] === Upgrades Infra/Transport API:: * Bump `TransportVersion` to the first non-release version number. Transport protocol is now versioned independently of release version. {es-pull}95286[#95286] Network:: * Upgrade Netty to 4.1.92 {es-pull}95575[#95575] * Upgrade Netty to 4.1.94.Final {es-pull}97112[#97112] Search:: * Upgrade Lucene to a 9.7.0 snapshot {es-pull}96433[#96433] * Upgrade to new lucene snapshot 9.7.0-snapshot-a8602d6ef88 {es-pull}96741[#96741]