This brings in the fixes from #130020, with minor fixes to address review
nits from that PR.
Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
Initial version of patterned_text mapper. Behaves similarly to match_only_text. This version uses a single SortedSetDocValues for a template and another for arguments. It splits the message by delimiters, the classifies a token as an argument if it contains a digit. All arguments are concatenated and inserted as a single doc value. A single inverted index is used, without positions. Phrase queries are still possible, using the SourceConfirmedTextQuery, but are not fast.
- loads the downloaded GeoIP databases from system index to ingest node file system for each project
- each project's databases are loaded to directory `tmp/geoip-databases/{nodeId}/{projectId}`
The vast majority of ingest pipelines are light CPU
operations. We don't want these to be put behind IO work on the write
executor. Instead, execute these on the coordination pool.
Occasional shard allocation issues were causing the YAML tests to fail
because the shard that had the document with the max timestamp in it
would be unavailable.
Fixes#118217
This change makes the GeoIp persistent task executor/downloader multi-project aware.
- the database downloader persistent task will be at the project level, meaning there will be a downloader instance per project
- persistent task id is prefixed with project id, namely `<project-id>/geoip-downloader` for cluster in MP mode
* Enable And Disable Endpoint
* Status Endpoint
* Integration Tests
* REST Spec
* REST Spec tests
* Some documentation
* Update docs/changelog/129474.yaml
* Fix failing security test
* PR Fixes
* PR Fixes - Add missing feature flag name to YAML spec
* PR Fixes - Fix support for timeout and master_timeout parameters
* PR Fixes - Make the REST handler validation happy with the new params
* Delete docs/changelog/129474.yaml
* PR Fixes - Switch to local metadata action type and improve request handling
* PR Fixes - Make enable / disable endpoint cancellable
* PR Fixes - Switch timeout param name for status endpoint
* PR Fixes - Switch timeout param name for status endpoint in spec
* PR Fixes - Enforce local only use for status action
* PR Fixes - Refactor StreamsMetadata into server
* PR Fixes - Add streams module to multi project YAML test suite
* PR Fixes - Add streams cluster module to multi project YAML test suite
Our hypothesis is that the path is not round-tripping on Windows. This
fixes the test by always using the canonical string for the path, when
inserting and when purging.
Fixes#129635
The stateful side tests generally do not exercise multi-project.
Therefore we need to use the default project-id instead of random one.
Relates: #127631
These usages had the potential of causing test failures when a data
stream was created before midnight and the backing index name generation
ran the next day - which would be millisecconds apart. To avoid these
failures, we update the tests to be robust to these time differences.
Resolves#123376
We added a new `projectClient` method on `Client` in #129174. We now
update the usages of the old method (on `ProjectResolver`) to use the
new one and we delete the old method.
This PR updates some of the existing checks for cluster blocks to also consider project global blocks. It adds a few project-aware flavour of existing methods in `ClusterBlocks`. `globalBlockedException(ClusterBlockLevel)` is the mostly used one. I've updated only some of the obvious ones here.
Follow up to https://github.com/elastic/elasticsearch/pull/127978
Relates ES-11209
We originally defined the `projectClient` method on `ProjectResolver` as
a convenience method to execute API calls for specific projects. That
method requires a reference to both a `ProjectResolver` and a `Client`.
We now introduce the same method directly on the `Client` interface and
inject a `ProjectResolver` there, removing the need for a
`ProjectResolver` reference in places that just want to execute API
requests on a specific project.
To reduce the number of changes, this change solely focuses on
introducing the new method. Future changes will migrate the uses of the
original method to the new one and remove the original altogether.
Don't store text and match_only_text field by default when source mode is synthetic and a field is a multi field or when there is a suitable multi field.
Without this change, ES would store field otherwise twice in a multi-field configuration.
For example:
```
...
"os": {
"properties": {
"name": {
"ignore_above": 1024,
"type": "keyword",
"fields": {
"text": {
"type": "match_only_text"
}
}
}
...
```
In this case, two stored fields were added, one in case for the `name` field and one for `name.text` multi-field.
This change prevents this, and would never store a stored field when text or match_only_text field is a multi-field.
This removes all non-test usages of
```
Metadata.Builder.putCustom(String type, ProjectCustom custom)
```
And replaces it with appropriate calls to the equivalent method on
`ProjectMetadata.Builder`.
In most cases this _does not_ make the code project aware, but does
reduce the number of deprecated methods in use.
Initially we added to the `include_defaults` to the get data stream
options REST API as it was used in the lifecycler API; however, we
decided to simplify it and not use it. We remove it now before it gets
adopted.
Last part size is wrongly computed to 0 when the last part's length is
exactly equal to the size of a part. Would have probably be caught by an
existing assertion.
Relates ES-11815
More robust test for closed clients holder. Also changes
IllegalStateException to AlreadyClosedException for both closed manager
and holder.
Resolves: #128707
With project secrets in place, we can now create and manage per-project
repository clients in addition to the cluster level repository clients.
This PR adds a manager class for that. Note that the logic is not yet
fully wired because it needs per-project repository/objec_store which
will be added in separate PRs (as part of MP snapshots and MP
objecStoreService). As such the tests are currently unit tests.
Relates: https://github.com/elastic/elasticsearch/pull/126584 Resolves:
ES-11713
