Rest client: don't reuse that same HttpAsyncResponseConsumer across multiple retries (#21378)

* Rest client: don't reuse that same HttpAsyncResponseConsumer across multiple retries

Turns out that AbstractAsyncResponseConsumer from apache async http client is stateful and cannot be reused across multiple requests. The failover mechanism was mistakenly reusing that same instance, which can be provided by users, across retries in case nodes are down or return 5xx errors. The downside is that we have to change the signature of two public methods, as HttpAsyncResponseConsumer cannot be provided directly anymore, rather its factory needs to be provided which is going to be used to create one instance of the consumer per request attempt.

Up until now we tested our RestClient against multiple nodes only in a mock environment, where we don't really send http requests. In that scenario we can verify that retries etc. work properly but the interaction with the http client library in a real scenario is different and can catch other problems. With this commit we also add an integration test that sends requests to multiple hosts, and some of them may also get stopped meanwhile. The specific test for pathPrefix was also removed as pathPrefix is now randomly applied by default, hence implicitly tested. Moved also a small test method that checked the validity of the path argument to the unit test RestClientSingleHostTests.

Also increase default buffer limit to 100MB and make it required in default consumer

The default buffer limit used to be 10MB but that proved not to be high enough for scroll requests (see reindex from remote). With this commit we increase the limit to 100MB and make it a bit more visibile in the consumer factory.

docs/java-rest/usage.asciidoc

@@ -117,7 +117,7 @@ Response performRequest(String method, String endpoint,
 Response performRequest(String method, String endpoint,
                         Map<String, String> params,
                         HttpEntity entity,
-                        HttpAsyncResponseConsumer<HttpResponse> responseConsumer,
+                        HttpAsyncResponseConsumerFactory responseConsumerFactory,
                         Header... headers)
     throws IOException;
 
@@ -141,7 +141,7 @@ void performRequestAsync(String method, String endpoint,
                          Map<String, String> params,
                          HttpEntity entity,
                          ResponseListener responseListener,
-                         HttpAsyncResponseConsumer<HttpResponse> responseConsumer,
+                         HttpAsyncResponseConsumerFactory responseConsumerFactory,
                          Header... headers);
 --------------------------------------------------
 
@@ -155,11 +155,12 @@ call (e.g. `/_cluster/health`)
 `params`:: the optional parameters to be sent as querystring parameters
 `entity`:: the optional request body enclosed in an
 `org.apache.http.HttpEntity` object
-`responseConsumer`:: the optional
+`responseConsumerFactory`:: the optional factory that is used to create an
 http://hc.apache.org/httpcomponents-core-ga/httpcore-nio/apidocs/org/apache/http/nio/protocol/HttpAsyncResponseConsumer.html[`org.apache.http.nio.protocol.HttpAsyncResponseConsumer`]
- callback. Controls how the response body gets streamed from a non-blocking
-HTTP connection on the client side. When not provided, the default
-implementation is used which buffers the whole response body in heap memory
+ callback instance per request attempt. Controls how the response body gets
+ streamed from a non-blocking HTTP connection on the client side. When not
+ provided, the default implementation is used which buffers the whole response
+ body in heap memory, up to 100 MB
 `responseListener`:: the listener to be notified upon asynchronous
 request success or failure
 `headers`:: optional request headers