This commit adds instrumentation for File methods that require read permissions.
see #122109 for the write side
(cherry picked from commit bf1cbb0e47)
# Conflicts:
# x-pack/plugin/security/src/main/plugin-metadata/entitlement-policy.yaml
Co-authored-by: Ryan Ernst <ryan@iernst.net>
Add missing entitlements for trust store if running in fips mode.
Fixes#122546, fixes#122569, fixes#122568, fixes#122680, fixes
#122566
(cherry picked from commit 87c58ff93f)
# Conflicts:
# muted-tests.yml
# x-pack/plugin/security/src/main/plugin-metadata/entitlement-policy.yaml
* Fix internal cluster and single node security tests (#121466)
This PR fixes SecuritySingleNodeTestCase and ProfileIntegTests tests.
- The security single node test failures are solved by ensuring every test starts with security index created and available. This is in order to have consistent state for every test. With the changes introduce in the #120323 PR, only the first test would execute with .security index being created async. Subsequent tests would execute without security index creation due to the fact that whole cluster is wiped after each test. This caused a flakiness only for the first test, because there was no mechanism in place to ensure that the .security index is active before test execution.
- The profile integration tests are solved by introducing an anonymous role which don't have application privileges. The application privileges are resolved from the .security index and assigned to all users, including the es_test_root user which is used during cluster wiping. Due to asynchronous nature of cluster setup and .security index creation, this now causes flakiness. The main problem is that wiping is done asynchronously and uses es_test_root which had assigned anonymous rac_role which depends on .security index being available for search in order to resolve application privileges. The application privilege resolution is done in buildRoleFromDescriptors which currently does not wait for security index availability(can be improved - but still wouldn't fix internal cluster tests). This wasn't a problem before just because we simply return empty results when .security index does not exist. There is some complexity in making internal clusters wait for availability of security shards before the test, so I think this solution is acceptable given that it's not required for this tests to have anonymous role with application privileges.
Resolves#121022Resolves#121096Resolves#121101Resolves#120988Resolves#121108Resolves#120983Resolves#120987Resolves#121179Resolves#121183Resolves#121346Resolves#121151Resolves#120985Resolves#121039Resolves#121483Resolves#121116Resolves#121258Resolves#121486
(cherry picked from commit 369c641302)
# Conflicts:
# muted-tests.yml
# x-pack/plugin/security/src/internalClusterTest/java/org/elasticsearch/xpack/security/authc/esnative/ReservedRealmElasticAutoconfigIntegTests.java
* fix compilation error
In test-scoped internal ITs the `cluster().assertAfterTest()` method was invoked
*after* the cluster nodes were closed. Consequently, the assertions that iterated
over the internal nodes (and asserted some state on nodes after the test) were
all effectively noops.
This PR reverses that order, so that after-test assertions are effective again.
When we detect that a Transform writes to the index and the index is
incompatible with the next version, change the message, detail, and
URL to help the user take the necessary steps to migrate the destination
index.
* [8.x] Logsdb and source only snapshots.
Backporting #122199 to 8.x branch.
Addresses a few issues with logsdb and source only snapshots:
* Avoid initializing index sorting, because sort fields will not have doc values.
* Also disable doc value skippers when doc values get disabled.
* As part of source only validation figure out what the nested parent field is.
Also added a few more tests that snapshot and restore logsdb data streams.
* fix test
* Add action to copy index metadata when reindexing data stream indices (#122535)
When reindexing data stream indices, parts of the index metadata needs to be copied from the source index to destination index, so that ILM and data stream lifecycle function properly. This adds a new CopyLifecycleIndexMetadataTransportAction which copies the following metadata from a source index to a destination index:
- creation date setting
- rollover info
- ILM custom metadata
(cherry picked from commit aba25c628a)
# Conflicts:
# x-pack/plugin/migrate/build.gradle
* remove timeouts which were not present in 8.18
* Optimize ST_EXTENT_AGG for geo_shape and cartesian_shape (#119889)
Support for `ST_EXTENT_AGG` was added in https://github.com/elastic/elasticsearch/pull/118829, and then partially optimized in https://github.com/elastic/elasticsearch/pull/118829. This optimization worked only for cartesian_shape fields, and worked by extracting the Extent from the doc-values and re-encoding it as a WKB `BBOX` geometry. This does not work for geo_shape, where we need to retain all 6 integers stored in the doc-values, in order to perform the datelline choice only at reduce time during the final phase of the aggregation.
Since both geo_shape and cartesian_shape perform the aggregations using integers, and the original Extent values in the doc-values are integers, this PR expands the previous optimization by:
* Saving all Extent values into a multi-valued field in an IntBlock for both cartesian_shape and geo_shape
* Simplifying the logic around merging intermediate states for all cases (geo/cartesian and grouped and non-grouped aggs)
* Widening test cases for testing more combinations of aggregations and types, and fixing a few bugs found
* Enhancing cartesian extent to convert from 6 ints to 4 ints at block loading time (for efficiency)
* Fixing bugs in both cartesian and geo extents for generating intermediate state with missing groups (flaky tests in serverless)
* Moved the int order to always match Rectangle for 4-int and Extent for 6-int cases (improved internal consistency)
Since the PR already changed the meaning of the invalid/infinite values of the intermediate state integers, it was already not compatible with the previous cluster versions. We disabled mixed-cluster testing to prevent errors as a result of that. This leaves us the opportunity to make further changes that are mixed-cluster incompatible, hence the decision to perform this consistency update now.
* Regenerate generated files
With #117410, each remote sink now has its own listener, and the main
query won't finish until all remote sink listeners have completed. As a
result, we no longer need to wait for the exchange source to finish.
This change removes the completion listener to simplify the exchange
service. The completion listener could previously return prematurely
while remote sinks were still being registered.
Closes#122408
There is a race condition where the test is trying to clean up while the
Transform auditor is still writing messages - resetting the plugin will
stop the auditor (and properly reset it).
Fix#12148
* Fix ReindexDataStreamIndexAction timestamp validation bug in tests (#122274)
Fix race condition test bugs related to the reindex-data-stream-pipeline. For tests that add doc without timestamp, then add mapping with timestamp, ensure green between adding doc and adding mapping. This makes sure that doc has been written to all shards and thus that timestamp validation does not occur while doc is being written to a shard. Delete pipeline in Before method, then wait for it to be re-created by the MigrateTemplateRegistry.
(cherry picked from commit 05a2003a9f)
# Conflicts:
# muted-tests.yml
# x-pack/plugin/migrate/src/internalClusterTest/java/org/elasticsearch/xpack/migrate/action/ReindexDatastreamIndexTransportActionIT.java
* getPipelines missing from 8.18
* [Transform] Delete Alias Write Index (#122074)
When the Transform is configured to write to an alias, specifying
`DELETE _transform/<id>?delete_dest_index` will follow the alias
to the concrete destination index.
Fix#121913
Co-authored-by: Przemysław Witek <przemyslaw.witek@elastic.co>
* Update for 8.x api
---------
Co-authored-by: Przemysław Witek <przemyslaw.witek@elastic.co>
Ensure that the old snapshot model_size_stats document is removed after the snapshot upgrade.
Marking it as a non-issue since the bug was not released yet.
This PR increases the waiting time for the DFA native process from 60 to 90 seconds to stabilize the flaky integration tests that sometimes fail because the job has not finished training within 60 seconds.
The nebula info broker plugin takes the information for the manifest from the java project settings rather then from
the compile task configuration. Instead of setting the compiler task configuration explicitly we now set the project
configuration accordingly. Also tweaked the javaTestCompile tasks to keep compiling with general minimum runtime version as we did before
(cherry picked from commit 6e6e42f5d4)
* Remove call to super.cleanupCluster in cleanup methods which does not override parent method (#122209)
ReindexDataStreamIndexAction.cleanupCluster called EsIntegTestCase.cleanupCluster, but did not override it. This caused EsIntegTestCase.cleanupCluster to be called twice, once in ReindexDataStreamIndexAction.cleanupCluster and once when the After annotation is called on EsIntegTestCase.
(cherry picked from commit 89ba03ecff)
# Conflicts:
# muted-tests.yml
# x-pack/plugin/migrate/src/internalClusterTest/java/org/elasticsearch/xpack/migrate/action/ReindexDatastreamIndexTransportActionIT.java
* deletePipeline not available in 8.x
* Adding condition to verify if the field belongs to an index
* Update docs/changelog/121720.yaml
* Remove unnecessary comma from yaml file
* remove duplicate inference endpoint creation
* updating isMetadata to return true if mapper has the correct type
* remove unnecessary index creation in yaml tests
* Adding check if the document has returned in the yaml test
* Updating test to skip time series check if index mode is standard
* Refactor tests to consider verifying every metafields with all index modes
* refactoring test to verify for all cases
* Adding assetFalse if not time_series and fields are from time_series
* updating test texts to have better description
The downsample task sometimes needs a little bit longer to complete so
we bump the timeout from 60s to 120s.
Fixes#122056
(cherry picked from commit 0ec2fe05ef)
# Conflicts:
# muted-tests.yml
When a node is shutting down, scheduling tasks for the Driver can result
in a rejection exception. In this case, we drain and close all
operators. However, we don't clear the pending tasks in the scheduler,
which can lead to a pending task being triggered unexpectedly, causing a
ConcurrentModificationException.
Since introducing the fail_fast (see #117410) option to remote sinks,
the ExchangeSource can propagate failures that can lead to circular
references. The issue occurs as follows:
1. remote-sink-1 fails with exception e1, and the failure collector collects e1.
2. remote-sink-2 fails with exception e2, and the failure collector collects e2.
3. The listener of remote-sink-2 propagates e2 before the listener of
remote-sink-1 propagates e1.
4. The failure collector in ExchangeSource sees [e1, e2] and suppresses
e2 to e1. The upstream sees [e2, e1] and suppresses e1 to e2, leading to
a circular reference.
With this change, we stop collecting failures in ExchangeSource.
Labelled this non-issue for an unreleased bug.
Relates #117410
The aggs timeout test waits for the agg to return and then double checks
that the agg is stopped using the tasks API. We're seeing some failures
where the tasks API reports that the agg is still running. I can't
reproduce them because computers. This adds two things:
1. Logs the hot_threads so we can see if the query is indeed still
running.
2. Retries the _tasks API for a minute. If it goes away soon after the
_search returns that's *fine*. If it sticks around for more than a
few seconds then the cancel isn't working. We wait for a minute
because CI can't be trusted to do anything quickly.
Closes#121993
It is possible to create an index in 7.x with a single type. This fixes the CreateIndexFromSourceAction to not copy that type over when creating a destination index from a source index with a type.
