Commit graph

25 commits

Author SHA1 Message Date
Wainer dos Santos Moschetta
908794db2d ci: fix run single specs
Some integration spec tests fail to run
individually due missing require modules.

This fix it by ensuring all specs and helpers
require the needed modules.

Fixes #7037

Fixes #7038
2017-05-06 01:30:34 +00:00
Pier-Hugues Pellerin
058cfc9312 Allow Logstash to accept a PROXY configuration
Logstash's plugin manager will now follow proxy configuration from the environment.
If you configure `http_proxy` and `https_proxy`, the manager will now use theses information for all the ruby http
connection and will also pass that information down to maven.

Fixes: #6619, #6528

Fixes #6825
2017-03-31 14:20:39 -04:00
Pier-Hugues Pellerin
2e908a0663 Allow to run the bin/logstash-plugin install <pack> outside of the Logstash directory
Fixes: #6599

Fixes #6602
2017-01-31 13:15:41 -05:00
Pier-Hugues Pellerin
1b812af23c Add bin/logstash-plugin prepare-offline-pack command
This new command replace the old workflow of `pack`, `unpack` and the `install --local`, and wrap all the logic into one uniform way of installing plugins.
The work is based on the flow developed for installing an x-pack inside Logstash, when you call prepare-offline-pack, the specified plugins and their dependencies will be packaged in a zip.
And this zip can be installed with the same flow as the pack.

Definition:

Source Logstash: Where you run the prepare-offline-pack.
Target Logstash: Where you install the offline package.

PROS:
- If you install a .gem in the source logstash, the .gem and his dependencies will be bundled.
- The install flow doesn't need to have access to the internet.
- Nothing special need to be setup in the target logstash environment.

CONS:
- The is one minor drawback, the plugins need to have their JARS bundled with them for this flow to work, this is currently the case for all the official plugins.
- The source Logstash need to have access to the internet when you install plugins before packaging them.

Usage examples:
bin/logstash-plugin prepare-offline-pack logstash-input-beats
bin/logstash-plugin prepare-offline-pack logstash-filter-jdbc logstash-input-beats
bin/logstash-plugin prepare-offline-pack logstash-filter-*
bin/logstash-plugin prepare-offline-pack logstash-filter-* logstash-input-beats

How to install:
bin/logstash-plugin install file:///tmp/logstash-offline-plugins-XXXX.zip

Fixes #6404
2017-01-03 13:59:51 -05:00
Tal Levy
f8b4c72a49 add queue stats to node/stats api (#6331)
* add queue stats to node/stats api

example queue section:

```
"queue" : {
    "type" : "persisted",
    "capacity" : {
      "page_capacity_in_bytes" : 262144000,
      "max_queue_size_in_bytes" : 1073741824,
      "max_unread_events" : 0
    },
    "data" : {
      "free_space_in_bytes" : 33851523072,
      "current_size_in_bytes" : 262144000,
      "storage_type" : "hfs",
      "path" : "/logstash/data/queue"
    },
    "events" : {
      "acked_count" : 0,
      "unread_count" : 0,
      "unacked_count" : 0
    }
}
```

Closes #6182.

* migrate to use period metric pollers for storing queue stats per pipeline
2016-12-15 13:19:08 -08:00
Suyog Rao
8cc356d2f9 Add feature flag support for RATS (#6328)
* Add feature flag support for RATS

Adds feature flag support for RATS integration tests. First feature to use it
is persistent_queues. You can enable a FF in travis by using the travis matrix
and adding a env variable called FEATURE_FLAG=<feature_name>.

To use feature flag, you need a logstash.yml file which has this feature turned on.
Unfortunately --path.settings needs an entire dir where logstash.yml is located, not
just the logstash.yml file, so we need to checkin the fixture which has the feature yml
and log4j.

For example, you can now set FEATURE_FLAG=persistent_queues which means travis will
run tests twice, one with this feature enabled and once without it. When the feature is enaabled
the logstash.yml located in qa/integration/fixtures/persistent_queues is picked up whenever LS is started in tests
2016-12-01 11:31:30 -08:00
Suyog Rao
93b91e90af Add more debuggability to travis
Fixes #6272
2016-11-17 14:22:23 -05:00
Pier-Hugues Pellerin
00e225a849 Feature: A way to install/remove a plugin pack
A pack in this context is a *bundle* of plugins that can be distributed outside of rubygems; it is similar to what ES and kibana are doing, and
the user interface is modeled after them. See https://www.elastic.co/downloads/x-pack

**Do not mix it with the `bin/logstash-plugin pack/unpack` command.**

- it contains one or more plugins that need to be installed
- it is self-contains with the gems and the needed jars
- it is distributed as a zip file
- the file structure needs to follow some rules.

- As a reserved name name on elastic.co download http server
    - `bin/plugin install logstash-mypack` will check on the download server if a pack for the current specific logstash version exist and it will be downloaded, if it doesn't exist we fallback on rubygems.
    - The file on the server will follow this convention `logstash-mypack-{LOGSTASH_VERSION}.zip`

- As a fully qualified url
    - `bin/plugin install http://test.abc/logstash-mypack.zip`, if it exists it will be downloaded and installed if it does not we raise an error.

- As a local file
    - `bin/plugin install file:///tmp/logstash-mypack.zip`, if it exists it will be installed

Fixes #6168
2016-11-17 14:00:02 -05:00
Suyog Rao
5534aa7cdb Fix node stats test
Fixes #6236
2016-11-11 11:52:19 -05:00
Suyog Rao
26bf1e4f21 Add JVM uptime stats
Fixes #6236
2016-11-11 11:52:19 -05:00
Suyog Rao
f2156bfcf8 add more debuggability for kafka tests
Fixes #6072
2016-10-19 20:50:09 -04:00
Suyog Rao
6d48896021 Use try instead of stud::try
Fixes #6031
2016-10-14 18:27:45 -04:00
Suyog Rao
9c5e56c0f3 Add new tests and some helpers
Fixes #6031
2016-10-14 18:27:45 -04:00
Suyog Rao
093ef92918 Make changes to test framework for new tests
Fixes #6031
2016-10-14 18:27:45 -04:00
Suyog Rao
3fa390e959 Add debug to Kafka scripts
Fixes #6054
2016-10-13 15:41:15 -04:00
Suyog Rao
4e33f66986 Improvements to RATS test suite
* Reduce sleeps and output messages
* Cache installed services directory

Fixes #5968
2016-10-05 11:26:18 -04:00
Suyog Rao
a79f08cf86 Enable RATS integration tests on travis
Fixes #5963
2016-09-23 12:00:14 -04:00
Pier-Hugues Pellerin
ad846339a2 fix beats integration test
Fixes #5917
2016-09-22 13:43:50 -04:00
Suyog Rao
886cd45cb0 changes after review
Fixes #5917
2016-09-22 13:43:50 -04:00
Suyog Rao
057302ad3a Fixes after @ph review
Fixes #5917
2016-09-22 13:43:50 -04:00
Pier-Hugues Pellerin
7ab30bd004 Update the configuration to make it work with the latest libbeat (#5944)
changes.

In beta1, some options changes occurred in filebeat with the name of the
ssl options.

Like theses:

tls=>ssl
certificate_key => key

Fixes #5917
2016-09-22 13:43:50 -04:00
Pier-Hugues Pellerin
8a4c5b4280 QA Integration dont ship with certificates (#5928)
We are removing the pre generated certificate from the repository, the
certificates needed for the test will be generated before downloading
filebeat.

Fixes #5917
2016-09-22 13:43:50 -04:00
Pier-Hugues Pellerin
87c6eaddb2 Adding integration testing for filebeat and make the test a bit more flexible
The test includes theses scenario

- No TLS
- TLS Server Auth
- TLS Mutual Auth

Theses theses test relies on a certicate include in the fixtures to correctly run,
there is also a script to generate them.

Make the ServiceLocator class is now more flexible and use the pattern matching
to create the registry on the fly

Fixes #5887

Fixes #5917
2016-09-22 13:43:50 -04:00
Suyog Rao
e7c542192a Cleanup of print messages
Fixes #5917
2016-09-22 13:43:49 -04:00
Suyog Rao
e008ddcdd2 Add initial round of integration tests
Fixes #5917
2016-09-22 13:43:49 -04:00