Enhance QA acceptance test framework docs by adding comments and and upgraded readme in order to provide more context and knowledge around it.

Fixes #5436

qa/README.md

@@ -4,7 +4,25 @@ Welcome to the acceptance test framework for logstash, in this small
 README we're going to describe it's features and the necessary steps you will need to
 follow to setup your environment.
 
-### Environment setup and Running Tests
+### Setup your environment
+
+In summary this test framework is composed of:
+
+* A collection of rspec helpers and matchers that make creating tests
+  easy.
+* This rspecs helpers execute commands over SSH to a set of machines.
+* The tests are run, for now, as vagrant (virtualbox provided) machines.
+
+As of this, you need to have installed:
+
+* The latest version vagrant (=> 1.8.1)
+* Virtualbox as VM provider (=> 5.0)
+
+Is important to notice that the first time you set everything up, or when a
+new VM is added, there is the need to download the box (this will
+take a while depending on your internet speed).
+
+### Running Tests
 
 It is possible to run the full suite of the acceptance test with the codebase by 
 running the command `ci/ci_acceptance.sh`, this command will generate the artifacts, bootstrap
@@ -18,10 +36,163 @@ details on how to install it.
 
 _Inside the `qa` directory_
 
-* Execute the command `bundle` this will pull the necessary dependencies in your environment.
-* start your machines with `bundle exec test:setup`
-* Run `rake test:ssh_config` to dump the ssh configuration to access the different vagrant machines, this will generate a file named `.vm_ssh_config` that is going to be used for the tests.
-* Run `bundle exec rake test:acceptance:all` to run all acceptance test
-  at once, there is also detailed tasks for platforms:
- * `rake test:acceptance:debian` for debian platforms.
- * `rake test:acceptance:centos` for centos platforms.
+First of all execute the command `bundle` this will pull the necessary
+dependencies in your environment, after this is done, this is the collection of task available for you:
+
+```
+skywalker% rake -T
+rake qa:acceptance:all              # Run all acceptance
+rake qa:acceptance:debian           # Run acceptance test in debian machines
+rake qa:acceptance:redhat           # Run acceptance test in redhat machines
+rake qa:acceptance:single[machine]  # Run one single machine acceptance test
+rake qa:acceptance:suse             # Run acceptance test in suse machines
+rake qa:vm:halt[platform]           # Halt all VM's involved in the acceptance test round
+rake qa:vm:setup[platform]          # Bootstrap all the VM's used for this tests
+rake qa:vm:ssh_config               # Generate a valid ssh-config
+```
+
+Important to be aware that using any of this commands:
+
+```
+rake qa:acceptance:all              # Run all acceptance
+rake qa:acceptance:debian           # Run acceptance test in debian machines
+rake qa:acceptance:redhat           # Run acceptance test in redhat machines
+rake qa:acceptance:suse             # Run acceptance test in suse machines
+```
+
+will bootstrap all selected machines. If you're willing to run on single
+platform you should use
+
+```
+rake qa:acceptance:single[machine]  # Run one single machine acceptance test
+```
+
+this will not do any bootstrap, so you are required to previously
+boostrap the VM yourself by doing `vagrant up`. This is like this
+because this command is only here for developers, not for automated
+CI's.
+
+
+### How to run tests
+
+In this framework we're using ssh to connect to a collection of Vagrant
+machines, so first and most important is to generate a valid ssh config
+file, this could be done running `rake qa:vm:ssh_config`. When this task
+is finished a file named `.vm_ssh_config` will be generated with all the
+necessary information to connect with the different machines.
+
+Now is time to run your test and to do that we have different options:
+
+* rake qa:acceptance:all              # Run all acceptance
+* rake qa:acceptance:debian           # Run acceptance test in debian machines
+* rake qa:acceptance:redhat           # Run acceptance test in redhat machines
+* rake qa:acceptance:suse             # Run acceptance test in suse machines
+* rake qa:acceptance:single[machine]  # Run one single machine acceptance test
+
+Generally speaking this are complex tests so they take a long time to
+finish completly, if you look for faster feedback see at the end of this
+README how to run fewer tests.
+
+## Architecture of the Framework
+
+If you wanna know more about how this framework works, here is your
+section of information.
+
+### Directory structure
+
+* ```acceptance/``` here it goes all the specs definitions.
+* ```config```  inside you can find all config files, for now only the
+  platform definition.
+* ```rspec``` here stay all framework parts necessary to get the test
+  running, you will find the commands, the rspec matchers and a
+collection of useful helpers for your test.
+* ```sys``` a collection of bash scripts used to bootstrap the machines.
+* ```vagrant``` classes and modules used to help us running vagrant.
+
+### The platform configuration file
+
+Located inside the config directory there is the platforms.json which is used to define the different platforms we test with.
+Important bits here are:
+
+* `latest` key defines the latest published version of LS release which is used to test the package upgrade scenario.
+* inside the `platforms` key you will find the list of current available
+  OS we tests with, this include the box name, their type and if they
+have to go under specific bootstrap scripts (see ```specific: true ```
+in the platform definition).
+
+This file is the one that you will use to know about differnt OS's
+testes, add new ones, etc..
+
+### I want to add a test, whad should I do?
+
+To add a test you basically should start by the acceptance directory,
+here you will find an already created tests, most important locations
+here are:
+
+* ```lib``` here is where the tests are living. If a test is not going
+  to be reused it should be created here.
+* ```shared_examples``` inside that directory should be living all tests
+  that could be reused in different scenarios, like you can see the CLI
+ones.
+
+but we want to write tests, here is an example of how do they look like,
+including the different moving parts we encounter in the framework.
+
+
+```
+  config = ServiceTester.configuration
+  config.servers.each do |address|
+    ##
+    # ServiceTester::Artifact is the component used to interact with the
+    # destination machineri and the one that keep the necessary logic
+    # for it.
+    ##
+
+    logstash = ServiceTester::Artifact.new(address, config.lookup[address])
+
+    ## your test code goes here.
+  end
+```
+
+this is important because as you know we test with different machines,
+so the build out artifact will be the component necessary to run the
+actions with the destination machine.
+
+but this is the main parts, to run your test you need the framework
+located inside the ```rspec``` directory. Here you will find a
+collection of commands, properly organized per operating system, that
+will let you operate and get your tests done. But don't freak out, we
+got all logic necessary to select the right one for your test.
+
+You'll probably find enough supporting classes for different platforms, but if not, feel free to add it.
+
+FYI, this is how a command looks like:
+
+```
+    def installed?(hosts, package)
+      stdout = ""
+      at(hosts, {in: :serial}) do |host|
+        cmd = sudo_exec!("dpkg -s  #{package}")
+        stdout = cmd.stdout
+      end
+      stdout.match(/^Package: #{package}$/)
+      stdout.match(/^Status: install ok installed$/)
+  end
+  ```
+this is how we run operations and wrap them as ruby code.
+
+### Running a test (detailed level)
+
+There is also the possibility to run your tests with more granularity by
+using the `rspec` command, this will let you for example run a single
+tests, a collection of them using filtering, etc.
+
+Check https://relishapp.com/rspec/rspec-core/v/3-4/docs/command-line for more details, but here is a quick cheat sheet to run them:
+
+# Run the examples that get "is installed" in their description
+
+*  bundle exec rspec acceptance/spec -e "is installed" 
+
+# Run the example desfined at line 11
+
+*  bundle exec rspec acceptance/spec/lib/artifact_operation_spec.rb:11

qa/acceptance/spec/lib/artifact_operation_spec.rb

@@ -4,6 +4,7 @@ require_relative '../shared_examples/installed'
 require_relative '../shared_examples/running'
 require_relative '../shared_examples/updated'
 
+# This tests verify that the generated artifacts could be used properly in a relase, implements https://github.com/elastic/logstash/issues/5070
 describe "artifacts operation" do
   config = ServiceTester.configuration
   config.servers.each do |address|

qa/acceptance/spec/lib/cli_operation_spec.rb

@@ -6,6 +6,8 @@ require_relative "../shared_examples/cli/logstash-plugin/list"
 require_relative "../shared_examples/cli/logstash-plugin/uninstall"
 require_relative "../shared_examples/cli/logstash-plugin/update"
 
+# This is the collection of test for the CLI interface, this include the plugin manager behaviour, 
+# it also include the checks for other CLI options.
 describe "CLI operation" do
   config = ServiceTester.configuration
   config.servers.each do |address|

qa/acceptance/spec/shared_examples/installed.rb

@@ -1,6 +1,7 @@
 require_relative '../spec_helper'
 require          'logstash/version'
 
+# This test checks if a package is possible to be installed without errors.
 RSpec.shared_examples "installable" do |logstash|
 
   before(:each) do

qa/acceptance/spec/shared_examples/running.rb

@@ -1,6 +1,7 @@
 require_relative '../spec_helper'
 require          'logstash/version'
 
+# Test if an installed package can actually be started and runs OK.
 RSpec.shared_examples "runnable" do |logstash|
 
   before(:each) do

qa/acceptance/spec/shared_examples/updated.rb

@@ -1,6 +1,7 @@
 require_relative '../spec_helper'
 require          'logstash/version'
 
+# This test checks if the current package could used to update from the latest version released.
 RSpec.shared_examples "updated" do |logstash|
 
   before (:all) { logstash.snapshot }

qa/platform_config.rb

@@ -2,9 +2,13 @@
 require "json"
 require "ostruct"
 
+# This is a wrapper to encapsulate the logic behind the different platforms we test with, 
+# this is done here in order to simplify the necessary configuration for bootstrap and interactions
+# necessary later on in the tests phases.
+#
 class PlatformConfig
 
-
+  # Abstract the idea of a platform, aka an OS
   class Platform
 
     attr_reader :name, :box, :type, :bootstrap

qa/rspec/commands.rb

@@ -12,6 +12,9 @@ require "forwardable"
 
 module ServiceTester
 
+  # An artifact is the component being tested, it's able to interact with
+  # a destination machine by holding a client and is basically provides all 
+  # necessary abstractions to make the test simple.
   class Artifact
 
     extend Forwardable
@@ -91,6 +94,8 @@ module ServiceTester
     end
   end
 
+  # Factory of commands used to select the right clients for a given type of OS and host name,
+  # this give you as much granularity as required.
   class CommandsFactory
 
     def self.fetch(type, host)