ESS support for FTR serverless tests. SSL support in kbn/es. kbn/es DX improvements. (#162673)

Closes #162593
Closes #163939 
Closes #162625

The original intention of this PR was to add FTR support for ESS.
However the scope increased as that also required adding SSL support due
to tests failing from disabled `security` and no authentication.
Additionally, after using serverless in `kbn/es` extensively for this,
there was a bit of friction in regards to DX.

## Summary
- Switch `x-pack/test_serverless` FTR to use ES serverless instead of
(stateful) snapshot
- Adds SSL support to Docker and Serverless in `kbn/es`
- Adds `port` option override
- Adds `teardown` option to kill running nodes if the process exits
without shutdown
- Adds `kill` option to kill running nodes on startup if detected
- Adds `--esFrom serverless` to FTR CLI
- Adds `files` option to mount extra files into containers
- For serverless, automatically attach to first node with `docker logs
-f es01` on startup for better DX.
- Added `background` flag to not attach `logs`.
- Adds graceful shutdown for ESS cluster
- Separate `docker pull` from `run` for better logging, ensures latest
image and stops multiple pulls of the same image occurring in parallel
- Align (most) default settings for ES serverless with `gradlew`
[settings](https://github.com/elastic/elasticsearch-serverless/blob/main/serverless-build-tools/src/main/kotlin/elasticsearch.serverless-run.gradle.kts#L8)
- Fixes Docker bind mount permissions in CI
- Fixes issue where `esFrom` would default to `snapshot` and override
FTR config settings.

### Checklist

- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

## Related Issues for Skipped Tests
Security Threat Hunting: #165135
Observability: #165138
Response Ops: #165145

---------

Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Patryk Kopycinski <contact@patrykkopycinski.com>

.buildkite/scripts/steps/functional/common.sh

@@ -21,3 +21,7 @@ if [[ -d "$cacheDir" ]]; then
 fi
 
 is_test_execution_step
+
+# logins into docker as a common step for functional tests
+echo "$KIBANA_DOCKER_PASSWORD" | docker login -u "$KIBANA_DOCKER_USERNAME" --password-stdin docker.elastic.co
+trap 'docker logout docker.elastic.co' EXIT

.buildkite/scripts/steps/functional/security_serverless_osquery.sh

@@ -2,10 +2,13 @@
 
 set -euo pipefail
 
-source .buildkite/scripts/common/util.sh
+source .buildkite/scripts/steps/functional/common.sh
 source .buildkite/scripts/steps/functional/common_cypress.sh
 
-.buildkite/scripts/bootstrap.sh
+# TODO: remove the line below to use build artifacts for tests.
+# in addition to remove the line, we will have to expose the kibana install dir into the downloaded build location
+# by exporting a var like:
+# export KIBANA_INSTALL_DIR=${KIBANA_BUILD_LOCATION}
 node scripts/build_kibana_platform_plugins.js
 
 export JOB=kibana-osquery-cypress-serverless