This commit updates the kafka setup scripts to ensure that the kafka setup is clean between builds, by
setting an explicit zookeeper data directory to be cleaned each time, and correctly overriding `log.dirs`
instead of `log.dir` to ensure that the kafka logs are written and wiped in a consistent place each time,
which helps when using the non-immutable images used in arm64 tests.
* Add feature flag support for RATS
Adds feature flag support for RATS integration tests. First feature to use it
is persistent_queues. You can enable a FF in travis by using the travis matrix
and adding a env variable called FEATURE_FLAG=<feature_name>.
To use feature flag, you need a logstash.yml file which has this feature turned on.
Unfortunately --path.settings needs an entire dir where logstash.yml is located, not
just the logstash.yml file, so we need to checkin the fixture which has the feature yml
and log4j.
For example, you can now set FEATURE_FLAG=persistent_queues which means travis will
run tests twice, one with this feature enabled and once without it. When the feature is enaabled
the logstash.yml located in qa/integration/fixtures/persistent_queues is picked up whenever LS is started in tests
