Route documents to the correct shards in tsdb (#77731)

* Route documents to the correct shards in tsdb

This causes elasticsearch to land documents from the same time series on
the same shard. It does so by adding a new index setting `routing_path`
which must be set when an index is in `mode: time_series` and may not be
set outside of that mode. That setting contains a list of patterns to
extract from the `_source` document that are hashed into the routing
value.

* Moar skip

* tsdb survives full cluster restart

* Remove auto generated id rejection

We do want to reject these documents but let's sae that for a follow up
change.

* Simplify

* Forbid routing_required

* Small

* Fork fork knife

* Let failures flow

* Fix full cluster

* Always fork

* Retry?

* Remove pressure test arm

* Add missing settings

* WIP

* Remove leftover

* More cleaning

* Fixup more tests

* Remove old skip

* New tests for Retry

* More tests

* Revert unrelated

* One dispatch please

* Stuff moved

* More moving

* Explain why fork

* Back to ActionRunnable

* Update comment

* Utility method

* Move routing_path under feature flag

* Imports

qa/multi-cluster-search/src/test/resources/rest-api-spec/test/multi_cluster/100_tsdb.yml

@@ -12,6 +12,7 @@ setup:
           settings:
             index:
               mode: time_series
+              routing_path: [metricset, k8s.pod.uid]
               number_of_replicas: 0
               number_of_shards: 2
           mappings:

qa/multi-cluster-search/src/test/resources/rest-api-spec/test/remote_cluster/10_basic.yml

@@ -166,6 +166,7 @@ tsdb:
           settings:
             index:
               mode: time_series
+              routing_path: [metricset, k8s.pod.uid]
               number_of_replicas: 0
               number_of_shards: 2
           mappings: