This change adds access to mapped match_only_text fields via the Painless scripting fields API. The
values returned from a match_only_text field via the scripting fields API always use source as described
by (#81246). These are not available via doc values so there are no bwc issues.
This change adds source fallback support for date and date_nanos by using the existing
SourceValueFetcherSortedNumericIndexFieldData to emulate doc values.
We have a `SingleMessageFiledInfo` defined in
`org/elasticsearch/xpack/core/ilm/step/info` to provide single message
info for `AsyncWaitStep` and `ClusterStateWaitStep`.But there are still
some steps like `CheckNotDataStreamWriteIndexStep` defining their own
single message info. This pr removes the duplicated info defination in
these steps and use `SingleMessageFiledInfo` instand.
The GetUserPrivileges API returns a 500 error when it is called with an
API key that has assigned role descriptors. This is because the
underlying LimitedRole class that represents the API key's effective
privileges does not support building a simple view of the privileges.
This PR changes the code to return 400 error instead of 500 along with a
better error message that suggests the GetApiKey API as an alternative.
Relates: #89058
This PR adds a new `errors` field in the ProfilehasPrivileges response
to report detailed errors encountered, including missing UIDs. It also
removes the existing `errors_uids` field since this is redundant after
the change.
This change adds access to mapped text fields via the Painless scripting fields API. The values returned
from a text field via the scripting fields API always use source as described by (#81246). Access via the
old-style through doc will still depend on field data, so there is no change and avoids bwc issues.
Currently, source fallback numeric types do not match doc values numeric types. Source fallback
numeric types de-duplicate numeric values in multi-valued fields. This change removes the de-
duplication for source fallback values for numeric types using value fetchers. This also adds test cases
for all the supported source fallback types to ensure they continue to match their doc values
counterparts exactly.
This change adds source fallback support for unsigned long by adding a new
SourceValueFetcherSortedUnsignedLongIndexFieldData similar to the other numeric types.
The documentation mentions the `noop` value ("Otherwise it does nothing (`noop`)"), however, the value used in the example script is `none`. This change corrects the value in the example script to `noop`.
This PR moved some logic that is only used by log in
IndexLifecycleRunner#isReadyToRansitionToThisPhase(...) into trace-log-
block which may save cpu and memory if trace log was not enabled.
This commit adds support for floating point node.processors setting.
This is useful when the nodes run in an environment where the CPU
time assigned to the ES node process is limited (i.e. using cgroups).
With this change, the system would be able to size the thread pools
accordingly, in this case it would round up the provided setting
to the closest integer.
When parsing mappings, we may find a field with same name specified twice, either
because JSON duplicate keys are allowed, or because a mix of object notation and dot
notation is used when providing mappings. The same can happen when applying dynamic
mappings as part of parsing an incoming document, as well as when merging separate
index templates that may contain the definition for the same field using a
mix of object notation and dot notation.
While we propagate the MapperBuilderContext across merge calls thanks to #86946, we do not
propagate the right context when we call merge on objects as part of parsing/building
mappings. This causes a situation in which the leaf fields that result from the merge
have the wrong path, which misses the first portion e.g. sub.field instead of obj.sub.field.
This commit applies the correct mapper builder context when building the object mapper builder
and two objects with same name are found.
Relates to #86946Closes#88573
Previously `graph` checked if the request timed out, then spent some
time doing work, then passed the timeout on to the next request. Over
and over again. It's quite possible that the response may not have timed
out for the first check but would have timed out for the second check.
This manifests as the timeout being sent to the next hop being a
negative number of milliseconds. We don't allow this sort of thing.
This fixes this by moving the timeout check to the same spot it is read
for setting the timeout on the next request - we just check if its `> 0`
to find the timeouts.
This does keep the request running slightly longer after it's officially
timed out - but it's just long enough to prepare the next layer of
request. Usually microseconds. Which should be fine.
Closes#55396
When calling RuntimeField.parseRuntimeFields() for fields defined in the
search request, we need to wrap the Map containing field definitions in another
Map that supports value removal, so that we don't inadvertently remove the
definitions from the root request. CompositeRuntimeField was not doing this
extra wrapping, which meant that requests that went to multiple shards and
that therefore parsed the definitions multiple times would throw an error
complaining that the fields parameter was missing, because the root request
had been modified.
This commit adds a short note to the 'search your data' docs around kNN search
to explain how approximate kNN works with aggregations:
* Make section on 'hybrid retrieval' more general and include aggregations info
* Remove an example response from the previous section on filtering, since this
page was getting long
This replaces the code that build `_id` in tsid indices that used to
re-parse the entire json object with one that reuses the parsed values.
It speed up writes by about 4%. Here's the rally output:
```
| Min Throughput | 8164.67 | 8547.24 | docs/s | +4.69% |
| Mean Throughput | 8891.11 | 9256.75 | docs/s | +4.11% |
| Median Throughput | 8774.52 | 9134.15 | docs/s | +4.10% |
| Max Throughput | 10246.7 | 10482.3 | docs/s | +2.30% |
```
I broke shard splitting when `_routing` is required and you use `nested`
docs. The mapping would look like this:
```
"mappings": {
"_routing": {
"required": true
},
"properties": {
"n": { "type": "nested" }
}
}
```
If you attempt to split an index with a mapping like this it'll blow up
with an exception like this:
```
Caused by: [idx] org.elasticsearch.action.RoutingMissingException: routing is required for [idx]/[0]
at org.elasticsearch.cluster.routing.IndexRouting$IdAndRoutingOnly.checkRoutingRequired(IndexRouting.java:181)
at org.elasticsearch.cluster.routing.IndexRouting$IdAndRoutingOnly.getShard(IndexRouting.java:175)
```
This fixes the problem by entirely avoiding the branch of code. That
branch was trying to find any top level documents that don't have a
`_routing`. But we *know* that there aren't any top level documents
without a routing in this case - the routing is "required". ES wouldn't
have let you index any top level documents without the routing.
This also adds a small pile of REST layer tests for shard splitting that
hit various branches in this area. For extra paranoia.
Closes#88109
Now that we're releasing synthetic _source as a tech preview feature, we
no longer want to remove the docs from the non-release builds. And we
want to mark all of the headings describing synthetic `_source` as a
preview.
Copies a few changelog edits from the 8.4 branch into main so that
people looking at the main branch see the same docs as people
looking at 8.4.
Relates #89078
Relates #89286
Relates #89376
A node can be informed that it is about to be shut down
using the node shutdown API. When this happens we
gracefully stop the jobs on the node and they persist
state. This state persistence includes latest quantiles,
and usually receipt of new quantiles triggers a
renormalization. However, in the case of an impending
node shutdown we do not want to be kicking off new
processes that may delay the shutdown.
This PR changes the anomaly detection job results
processor so that it will not trigger a renormalization
based on quantiles received after a node shutdown message
is received.
The docs for synthetic `_source` incorrectly claimed that synthetic
`_source` deduplicates numbers. It doesn't. The example below the prose
shows it *not* removing duplicates.
We have a check that enforces the total number of fields needs to be below a
certain (configurable) threshold. Before runtime fields did not contribute
to the count. This patch makes all runtime fields contribute to the
count, runtime fields:
- that were explicitly defined in mapping by a user
- as well as runtime fields that were dynamically created by dynamic
mappings
Closes#88265
Adds a simple preflight check to `JoinHelper#sendJoinRequest` to avoid
sending a join request if it looks like the `inflight_requests` circuit
breaker is going to trip on the join validation message.
Closes#85003
It's possible for a cluster state update task to emit deprecation
warnings, but if the task is executed in a batch then these warnings
will be exposed to the listener for every item in the batch. With this
commit we introduce a mechanism for tasks to capture just the warnings
relevant to them, along with assertions that warnings are not
inadvertently leaked back to the master service.
Closes#85506
Today if the GCS credentials file setting is invalid we report some kind
of JSON parsing error but it's not clear what JSON is being parsed so
the error is hard to track down. This commit adds the problematic
setting name to the exception message.
This PR builds on #86524, #87482, and #87306 by supporting the case where there has been no
master node in the last 30 second, no node has been elected master, and the current node is not
master eligible.
There are some redundant words so I just removed those words. Please accept this change.
(cherry picked from commit e1e5398051)
Co-authored-by: Adnan Ashraf <adnan.ashraff1@gmail.com>
An API key's effective permission is an intersection between its
assigned role descriptors and a snapshot of its owner user's role
descriptors (limited-by role descriptors). In #89166, the assigned role
descriptors are now returned by default in Get/Query API key responses.
This PR further adds support to optionally return limited-by role
descriptors in the responses. Unlike assign role descriptors, an API key
cannot view any limited-by role descriptors unless it has manage_api_key
or higher privileges.
Relates: #89058
The reloading secure settings action sends one node level request with
password (secureString) to each node. These node level requests share
the same secureString instance. This is not a problem when the requests
are sent across the wire because the secureString will end up to be
independent instances after de/serilization. But when the request is
handled by the local node, it skips the de/serialization process. This
means when the secureString gets closed, it is closed for all the node
level requests. If a node level request has not been sent across wire
when the secureString is closed under it, the serialization process will
result into error.
This PR fixes the bug by letting each Node level request creates a clone
of the secureString and have the Nodes level request to track all Node
level requests. All copies of secureString (on the coordinate node) will
be closed at Nodes request level which is safe because it is after
completion of all Node level requests.
Resolves: #88887
* [DOCS] Warn only one date format is added to the field date formats
When using multiple options in `dynamic_date_formats`, only one of the formats of the first document having a date matching one of the date formats provided will be used.
E.g.
```
PUT my-index-000001
{
"mappings": {
"dynamic_date_formats": [ "yyyy/MM", "MM/dd/yyyy"]
}
}
PUT my-index-000001/_doc/1
{
"create_date": "09/25/2015"
}
```
The generated mappings will be:
```
"mappings": {
"dynamic_date_formats": [
"yyyy/MM",
"MM/dd/yyyy"
],
"properties": {
"create_date": {
"type": "date",
"format": "MM/dd/yyyy"
}
}
},
```
Indexing a document with `2015/12` would lead to the `format` `"yyyy/MM"` being used for the `create_date`.
This can be misleading especially if the user is using multiple date formats on the same field.
The first document will determine the format of the `date` field being detected.
Maybe we should provide an additional example, such as:
```
PUT my-index-000001
{
"mappings": {
"dynamic_date_formats": [ "yyyy/MM||MM/dd/yyyy"]
}
}
```
My wording is not great, so feel free to amend/edit.
* Update docs/reference/mapping/dynamic/field-mapping.asciidoc
Reword and add code example
* Turned discussion of the two syntaxes into an admonition
* Fix failing tests
Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
The docs for `transport.ping_schedule` note that the transport client
defaults to a 5s ping schedule, but this is no longer relevant. This
commit drops this from the docs, and also moves the docs for this
setting further down the page to reflect its relative unimportance.
Today if a node shutdown is stalled due to unmoveable shards then we say
to use the allocation explain API to find details. In fact, since #78727
we include the allocation explanation in the response already so we
should tell users just to look at that instead. This commit adjusts the
message to address this.
This PR updates the URLs for several references that are being
used in the plugin document.
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Today if there are multiple nodes with the same name then
`POST /_cluster/voting_config_exclusions?node_names=ambiguous-name` will
return a `500 Internal Server Error` and a mysterious message. This
commit changes the behaviour to throw an `IllegalArgumentException`
(i.e. `400 Bad Request`) along with a more useful message describing the
problem.
