Commit graph

96 commits

Author SHA1 Message Date
Andrew Cholakian
036ad57d8b Run offline specs separately from online ones.
The internet disabling C hack can break other tests, so, we now run
those tests separately.

Fixes https://github.com/elastic/logstash/issues/7178

Fixes #7179
2017-05-22 15:28:27 +00:00
Joao Duarte
80303d2b32 add ExpectationNotMetError to Stud.try on monitoring_api_spec
in some tests against metrics api it's possible the values aren't there,
making the assertion fail. By default Stud.try only catches StandardError,
but rspec throws a RSpec::Expectations::ExpectationNotMetError on a failed
assertion, and this exception inherits directly from Exception.

This commit explicitly adds this exception to the list of Stud.try exceptions

Fixes #7177
2017-05-22 13:46:31 +00:00
Joao Duarte
08da5835d5 protect monitoring_api_spec against timing issues
Fixes #7145
2017-05-21 18:52:15 +00:00
Joao Duarte
22051d5db3 fix failing test due to faulty api request
fixes https://github.com/elastic/logstash/issues/7086

Fixes #7144
2017-05-18 11:00:27 +00:00
Pier-Hugues Pellerin
7cf1f90066 Relax the Geoip filter test
Since the database of the plugin can be update we cannot do a strict
assert on the geoip lat/long values instead we will use a range of
possible and valid latitude and longitude.

Fixes: #7119

Fixes #7122
2017-05-16 17:36:34 +00:00
Joao Duarte
55ad9850c8 fix remaining race condition in monitoring api integration spec
* add NoMethodError to list of allowed exceptions in try
2017-05-11 18:24:16 +01:00
Joao Duarte
e73a4cb209 make integration specs a bit more resilient
* wait for logstash starting in monitoring_api integration tests
* remove initialization code from monitoring api integration spec
2017-05-11 18:24:11 +01:00
Joao Duarte
ede2ca91fc fix monitoring api integration race condition
because the metric subsystem may not be ready yet when the tests ask for
the values, the test code may generate a NoMethodError accessing nested
hashes.

The NoMethodError exception aborts the try mechanism instead of retrying.

This PR rescues potential exceptions and adds more expectations to
protect the try block.
2017-05-11 14:17:03 +01:00
Wainer dos Santos Moschetta
908794db2d ci: fix run single specs
Some integration spec tests fail to run
individually due missing require modules.

This fix it by ensuring all specs and helpers
require the needed modules.

Fixes #7037

Fixes #7038
2017-05-06 01:30:34 +00:00
Josh Soref
df57e410cc Spelling fixes (#6806)
Spelling fixes across the board.
2017-04-04 12:07:16 -07:00
Pier-Hugues Pellerin
058cfc9312 Allow Logstash to accept a PROXY configuration
Logstash's plugin manager will now follow proxy configuration from the environment.
If you configure `http_proxy` and `https_proxy`, the manager will now use theses information for all the ruby http
connection and will also pass that information down to maven.

Fixes: #6619, #6528

Fixes #6825
2017-03-31 14:20:39 -04:00
Joao Duarte
c5b7cbeacc introduce locking in path.data
Fixes #6738
2017-02-24 05:27:10 -05:00
Pier-Hugues Pellerin
5f13c3fee3 Fix the multiple instance on the same machine test for when running under the PQ
This PR make the smoke test for the multiple instances of logstash works
and also change the assertions of a few tests to make it more robust.

Instead of a `try` I assert the creation of a file generated by a fixed generator.

Fixes: #6707

Fixes #6709
2017-02-15 13:07:41 -05:00
Pier-Hugues Pellerin
2e908a0663 Allow to run the bin/logstash-plugin install <pack> outside of the Logstash directory
Fixes: #6599

Fixes #6602
2017-01-31 13:15:41 -05:00
Tal Levy
bca87bae40 remove current_size_in_bytes and acked info from node stats
re #6508.

- removed `acked_count`, `unacked_count`, and migrated `unread_count` to
top-level `events` field.
- removed `current_size_in_bytes` info from queue node stats

Fixes #6510
2017-01-09 19:19:28 -05:00
Pier-Hugues Pellerin
1b812af23c Add bin/logstash-plugin prepare-offline-pack command
This new command replace the old workflow of `pack`, `unpack` and the `install --local`, and wrap all the logic into one uniform way of installing plugins.
The work is based on the flow developed for installing an x-pack inside Logstash, when you call prepare-offline-pack, the specified plugins and their dependencies will be packaged in a zip.
And this zip can be installed with the same flow as the pack.

Definition:

Source Logstash: Where you run the prepare-offline-pack.
Target Logstash: Where you install the offline package.

PROS:
- If you install a .gem in the source logstash, the .gem and his dependencies will be bundled.
- The install flow doesn't need to have access to the internet.
- Nothing special need to be setup in the target logstash environment.

CONS:
- The is one minor drawback, the plugins need to have their JARS bundled with them for this flow to work, this is currently the case for all the official plugins.
- The source Logstash need to have access to the internet when you install plugins before packaging them.

Usage examples:
bin/logstash-plugin prepare-offline-pack logstash-input-beats
bin/logstash-plugin prepare-offline-pack logstash-filter-jdbc logstash-input-beats
bin/logstash-plugin prepare-offline-pack logstash-filter-*
bin/logstash-plugin prepare-offline-pack logstash-filter-* logstash-input-beats

How to install:
bin/logstash-plugin install file:///tmp/logstash-offline-plugins-XXXX.zip

Fixes #6404
2017-01-03 13:59:51 -05:00
Tal Levy
f8b4c72a49 add queue stats to node/stats api (#6331)
* add queue stats to node/stats api

example queue section:

```
"queue" : {
    "type" : "persisted",
    "capacity" : {
      "page_capacity_in_bytes" : 262144000,
      "max_queue_size_in_bytes" : 1073741824,
      "max_unread_events" : 0
    },
    "data" : {
      "free_space_in_bytes" : 33851523072,
      "current_size_in_bytes" : 262144000,
      "storage_type" : "hfs",
      "path" : "/logstash/data/queue"
    },
    "events" : {
      "acked_count" : 0,
      "unread_count" : 0,
      "unacked_count" : 0
    }
}
```

Closes #6182.

* migrate to use period metric pollers for storing queue stats per pipeline
2016-12-15 13:19:08 -08:00
Suyog Rao
678ebd1efc Add reload stats at the instance level
Reload stats are currently reported at the pipeline level. The instance
level aggregates these stats across the pipelines

Fixes #6350

Fixes #6367
2016-12-13 00:59:24 -05:00
Suyog Rao
9d1da2f0d6 fix failing RATS test 2016-12-01 12:21:43 -08:00
Suyog Rao
8cc356d2f9 Add feature flag support for RATS (#6328)
* Add feature flag support for RATS

Adds feature flag support for RATS integration tests. First feature to use it
is persistent_queues. You can enable a FF in travis by using the travis matrix
and adding a env variable called FEATURE_FLAG=<feature_name>.

To use feature flag, you need a logstash.yml file which has this feature turned on.
Unfortunately --path.settings needs an entire dir where logstash.yml is located, not
just the logstash.yml file, so we need to checkin the fixture which has the feature yml
and log4j.

For example, you can now set FEATURE_FLAG=persistent_queues which means travis will
run tests twice, one with this feature enabled and once without it. When the feature is enaabled
the logstash.yml located in qa/integration/fixtures/persistent_queues is picked up whenever LS is started in tests
2016-12-01 11:31:30 -08:00
Pier-Hugues Pellerin
4f5f3490aa Relax the acceptance test expectation to match only the exception message
Fixes: #6310

Fixes #6325
2016-11-29 14:24:37 -05:00
Pier-Hugues Pellerin
77255be1fc use a temporary path instead of a temporary file
Using an empty file for the regitry make FB quits with a EOF error

Fixes: #6285

Fixes #6286
2016-11-21 20:10:54 -05:00
Suyog Rao
93b91e90af Add more debuggability to travis
Fixes #6272
2016-11-17 14:22:23 -05:00
Pier-Hugues Pellerin
00e225a849 Feature: A way to install/remove a plugin pack
A pack in this context is a *bundle* of plugins that can be distributed outside of rubygems; it is similar to what ES and kibana are doing, and
the user interface is modeled after them. See https://www.elastic.co/downloads/x-pack

**Do not mix it with the `bin/logstash-plugin pack/unpack` command.**

- it contains one or more plugins that need to be installed
- it is self-contains with the gems and the needed jars
- it is distributed as a zip file
- the file structure needs to follow some rules.

- As a reserved name name on elastic.co download http server
    - `bin/plugin install logstash-mypack` will check on the download server if a pack for the current specific logstash version exist and it will be downloaded, if it doesn't exist we fallback on rubygems.
    - The file on the server will follow this convention `logstash-mypack-{LOGSTASH_VERSION}.zip`

- As a fully qualified url
    - `bin/plugin install http://test.abc/logstash-mypack.zip`, if it exists it will be downloaded and installed if it does not we raise an error.

- As a local file
    - `bin/plugin install file:///tmp/logstash-mypack.zip`, if it exists it will be installed

Fixes #6168
2016-11-17 14:00:02 -05:00
Suyog Rao
5b21ca10ff shutdown after spec
Fixes #6236
2016-11-11 11:52:19 -05:00
Suyog Rao
5534aa7cdb Fix node stats test
Fixes #6236
2016-11-11 11:52:19 -05:00
Suyog Rao
26bf1e4f21 Add JVM uptime stats
Fixes #6236
2016-11-11 11:52:19 -05:00
Andrew Cholakian
efefe7f32a Add Persistent UUID for Agent
This adds two new fields 'id', and 'name' to the base metadata for API requests.
These fields are now returned at all API endpoints by default.

The `id` field is the persisted UUID, the name field is the custom name
the user has passed in (defaulted to the hostname).

I renamed `node_uuid` and `node_name` to just `id` and `name` to be
inline with Elasticsearch.

This also fixed a broken test double in `webserver_spec.rb` that was
using doubles across threads which created hidden errors.

Fixes #6224
2016-11-09 07:40:47 -05:00
Pier-Hugues Pellerin
4f5146536d Make sure we have we configure the ssl_certificate_authorities and set the verify mode to force_peer to make sure the input uses it.
Fixes #6185

Fixes #6192
2016-11-04 13:21:26 -04:00
Tal Levy
6eef09de32 slowlog
Fixes #6128
2016-11-02 18:48:40 -04:00
Suyog Rao
1672819acc Fix failing travis test
Fixes #6176
2016-11-02 18:10:39 -04:00
Suyog Rao
22374fdefc Make logstash.yml optional
5.0.0 required Logstash to have a valid logstash.yml before it could start successfully. This
was mostly fine for users who installed Logstash via tar.gz, but many many folks who install
it via packages still start Logstash manually. Also, our documentation uses -e flag for
getting started on Logstash and sending their first event.logstash.yml has only defaults defined,
and there is no required parameter to start Logstash. We should be able to use the defaults if no
logstash.yml. Obviously, this is not ideal from a user point of view, so we should log a warning but
continue to bootstrap.

Fixes #6170

Fixes #6172
2016-11-02 15:51:59 -04:00
Suyog Rao
de5c23cb0a -e and -f should merge even without valid conf file
Previously if both -e and -f was specified, LS required that
-f still have valid config file(s) before merging. This fixes it
to either have one of -f or -e provided

Fixes #6164
2016-11-01 18:39:50 -07:00
Suyog Rao
f2156bfcf8 add more debuggability for kafka tests
Fixes #6072
2016-10-19 20:50:09 -04:00
Pere Urbon-Bayes
f84cda02e3 fix suse installed? command and the suse-12 openjdk8 install procedure
Fixes #5861
2016-10-17 21:44:27 -04:00
Suyog Rao
eda291b5be refine tests
Fixes #6031
2016-10-14 18:27:45 -04:00
Suyog Rao
645b796c2b Make tests more robust
Fixes #6031
2016-10-14 18:27:45 -04:00
Suyog Rao
4e716d9dca Increase the tries
Fixes #6031
2016-10-14 18:27:45 -04:00
Suyog Rao
6d48896021 Use try instead of stud::try
Fixes #6031
2016-10-14 18:27:45 -04:00
Suyog Rao
fd07eb185d Use randomized ports
Fixes #6031
2016-10-14 18:27:45 -04:00
Suyog Rao
9c5e56c0f3 Add new tests and some helpers
Fixes #6031
2016-10-14 18:27:45 -04:00
Suyog Rao
093ef92918 Make changes to test framework for new tests
Fixes #6031
2016-10-14 18:27:45 -04:00
Suyog Rao
7731fc09ca Add new integration tests
Fixes #6031
2016-10-14 18:27:45 -04:00
Suyog Rao
3fa390e959 Add debug to Kafka scripts
Fixes #6054
2016-10-13 15:41:15 -04:00
Suyog Rao
b9c45083ba Fix acceptance tests
Fixes #6051
2016-10-13 11:51:14 -04:00
Suyog Rao
9c5e558c5a Deprecate uninstall for Plugin Manager
Adds remove and deprecates uninstall

Fixes #6041

Fixes #6042
2016-10-13 10:12:23 -04:00
fanfan
f51736dc0d typo fix for qa/README
Fixes #6033
2016-10-12 09:06:13 -04:00
yangwei71
a9547d4f02 fix a wrong word
the wrong word in this sentence "Run the example desfined at line 11" .
“desfined” change to "defined".

Fixes #5987
2016-10-11 12:39:23 -04:00
Suyog Rao
4e33f66986 Improvements to RATS test suite
* Reduce sleeps and output messages
* Cache installed services directory

Fixes #5968
2016-10-05 11:26:18 -04:00
Suyog Rao
a79f08cf86 Enable RATS integration tests on travis
Fixes #5963
2016-09-23 12:00:14 -04:00