Commit graph

10 commits

Author SHA1 Message Date
Armin
982c44bc1f PERFORMANCE: LIR Pipeline Execution
Fixes #8357
2017-10-05 20:37:25 +00:00
Jake Landis
d2011f9185 Test fixes: This should fix a couple different test failures.
* Fixes #8344 - Ruby Unit Test can hang with no information as to why
* Fixes #7227 - MetricStore::MetricNotFound
* Fixes #8204 - `config.reload.automatic` is set to`FALSE

I am only certain it fixes #8344 and #7227, since I can not reproduce #8204.

Fixes #8376
2017-09-24 03:39:29 +00:00
Andrew Cholakian
f5c6c5a4b1 Use file hashing where possible in LIR
This commit further improves hashing performance, using SourceWithMetadata
objects where present to determine Vertex IDs and the hash of the source
file. If those objects are not present LIR will revert to graph hashing,
which is slower, but always works. This is still useful for things like
testing, so it makes sense to leave that code in.

In the future it might be nice to extract the hashing code using the
strategy pattern to clean things up.

This commit also improves the Hashable interface, moving hashSource into
its own interface, letting classes that do their own hashing simply
implement uniqueHash instead of requiring them to also implement
hashSource.

A requirement of these changes was also that SourceWithMetadata no
longer accept null or empty arguments. This patch now enforces that
requirement. This also is nicer from an API standpoint, as you now know
that if a SourceWithMetadata object exists, it will actually have all
its fields in use.

Fixes #7408
2017-06-13 14:09:17 +00:00
Joao Duarte
bed8b8a084 support multiple pipelines in one logstash instance
* add multi_local source for multi pipelines
* introduce pipelines.yml
* introduce PipelineSettings class
* support reloading of pipeline parameters
* fix pipeline api call for _node/pipelines
* inform user pipelines.yml is ignored if -e or -f is enabled
2017-05-30 09:47:53 +01:00
Andrew Cholakian
78f70634ed Refactor API specs to play well with others
This is a big chang, it:

1. Moves API specs out of their special hierarchy
2. Removes the API spec spec_helper
3. Reactivates that stats command spec (that was accidentally not being
run before due to it not having _spec as a suffix

This was required to fix the preceeding commit, where we added a
before(:each) hook to the spec_helper that wasn't being picked up in
some cases due to the existence of two spec helpers and a $LOAD_PATH
that could change.

Fixes #7132
2017-05-18 13:45:27 +00:00
Andrew Cholakian
28e51f5c6a Unify ConfigPart and SourceMeta into SourceWithMetadata
This unifies the two different config classes that represented mainly
the same data. While this does expose a plain java class into ruby
this works fine because ruby only needs to access and set values, not
work with ruby return types.

Fixes #7003

Fixes #7004
2017-05-02 13:31:06 +00:00
Pier-Hugues Pellerin
645fcec234 Refactor of the Agent class and the loading of the configuration
This PR introduces majors changes inside Logstash, to help build future
features like supporting multiple pipeline or java execution.

The previous implementation of the agent made the class hard to refactor or add new feature to it, because it needed to
know about too much internal of the pipeline and how the configuration existed.

The changes includes:
- Externalize the loading of the configuration file using a `SourceLoader`
- The source loader can support multiple sources and will aggregate them.
- We keep some metadata about the original file so the LIR can give better feedback.
- The Agent now ask the `SourceLoader` to know which configuration need to be run
- The Agent now uses a converge state strategy to handle: start, reload, stop
- Each actions executed on the pipeline are now extracted into their own classes to help with migration to a new pipeline execution.
- The pipeline now has a start method that handle the tread
- Better out of the box support for multiple pipeline (monitoring)
- Refactor of the spec of the agent

Fixes #6632
2017-04-14 11:42:00 +00:00
Andrew Cholakian
76711a4785 LIR UI (Read Only) (#6241)
* Read-only Java IR
* Consistent ID generation as opposed to UUIDs

Fixes #6708
2017-04-03 08:49:00 -04:00
Andrew Cholakian
80d21712c6 Add Persistent UUID for Agent
This adds two new fields 'id', and 'name' to the base metadata for API requests.
These fields are now returned at all API endpoints by default.

The `id` field is the persisted UUID, the name field is the custom name
the user has passed in (defaulted to the hostname).

I renamed `node_uuid` and `node_name` to just `id` and `name` to be
inline with Elasticsearch.

This also fixed a broken test double in `webserver_spec.rb` that was
using doubles across threads which created hidden errors.

Fixes #6224
2016-11-09 07:40:47 -05:00
Pier-Hugues Pellerin
09a4d35687 Silence PUMA errors when an exception occur in logstash core
When Logstash encounter an exception Puma was getting really noisy about
the file descriptor it used. Its because Puma has an infinite loop into
his reactor and when logstash is dying from the exception the FD get
removed under the puma threads.

Puma is using the constants `STDERR` and `STDOUT` to give information
about the errors, there is no easy way to provide Puma with a custom IO.
This PR hacks the Puma name to redefine both constants and send the
information to the logger using the `debug` level.

Also when we start the server we can pass a custom Puma::Events instance
to record some of the errors happening in the internal threads, this PR
also make sure we send all theses message to the configured logger.

Fixes: #5822

Fixes #5869
2016-09-08 09:39:45 -04:00