Fix acceptance/packaging upgrade test near a release (#15826) (#15829)

The current mechanism of discovering the latest released version per
branch (via ARTIFACTS_API) isn't foolproof near the time of a new
release, as it may be pick a version that hasn't been released
yet. This leads to failures[^1] of the packaging upgrade tests, as we
attempt to download a package file that doesn't exist yet.

This commit switches to an API that that is more up to date regarding
the release version truth.

[^1]: https://buildkite.com/elastic/logstash-exhaustive-tests-pipeline/builds/125#018d319b-9a33-4306-b7f2-5b41937a8881/1033-1125

(cherry picked from commit 15e19a96c2)

Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>

qa/acceptance/helpers.rb

@@ -16,9 +16,8 @@
 # under the License.
 
 require 'net/http'
-require 'json'
 
-ARTIFACTS_API = "https://artifacts-api.elastic.co/v1/versions"
+ARTIFACT_VERSIONS_API = "https://storage.googleapis.com/artifacts-api/releases.properties"
 
 def logstash_download_metadata(version, arch, artifact_type)
   filename = "logstash-#{version}-#{arch}.#{artifact_type}"
@@ -26,14 +25,12 @@ def logstash_download_metadata(version, arch, artifact_type)
 end
 
 def fetch_latest_logstash_release_version(branch)
-  uri = URI(ARTIFACTS_API)
+  uri = URI(ARTIFACT_VERSIONS_API)
+  major = branch.split('.').first
 
   response = retryable_http_get(uri)
-  versions_data = JSON.parse(response)
 
-  filtered_versions = versions_data["versions"].select { |v| v.start_with?(branch) && !v.include?('SNAPSHOT') }
-
-  return filtered_versions.max_by { |v| Gem::Version.new(v) }
+  response.match(/current_#{major}=(\S+)/)&.captures&.first
 end
 
 def retryable_http_get(uri, max_retries=5, retry_wait=10)