github-mirrors/logstash
1
0
Fork 0
mirror of https://github.com/elastic/logstash.git synced 2025-04-24 22:57:16 -04:00
Code Issues Projects Releases Packages Wiki Activity

Feat: x-pack cloud id/auth for monitoring/management (#11496)

Browse source 
resolves #11488
This commit is contained in:
Karol Bucek 2020-01-14 21:35:32 +01:00 • committed by GitHub
parent 8367abb6cd
commit 3a1194edc5
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
11 changed files with 224 additions and 17 deletions
Download patch file Download diff file Expand all files Collapse all files

6
config/logstash.yml
View file

@ -224,6 +224,9 @@
#xpack.monitoring.elasticsearch.username: logstash_system
#xpack.monitoring.elasticsearch.password: password
#xpack.monitoring.elasticsearch.hosts: ["https://es1:9200", "https://es2:9200"]
# an alternative to hosts + username/password settings is to use cloud_id/cloud_auth
#xpack.monitoring.elasticsearch.cloud_id: monitoring_cluster_id:xxxxxxxxxx
#xpack.monitoring.elasticsearch.cloud_auth: logstash_system:password
#xpack.monitoring.elasticsearch.ssl.certificate_authority: [ "/path/to/ca.crt" ]
#xpack.monitoring.elasticsearch.ssl.truststore.path: path/to/file
#xpack.monitoring.elasticsearch.ssl.truststore.password: password
@ -241,6 +244,9 @@
#xpack.management.elasticsearch.username: logstash_admin_user
#xpack.management.elasticsearch.password: password
#xpack.management.elasticsearch.hosts: ["https://es1:9200", "https://es2:9200"]
# an alternative to hosts + username/password settings is to use cloud_id/cloud_auth
#xpack.management.elasticsearch.cloud_id: management_cluster_id:xxxxxxxxxx
#xpack.management.elasticsearch.cloud_auth: logstash_admin_user:password
#xpack.management.elasticsearch.ssl.certificate_authority: [ "/path/to/ca.crt" ]
#xpack.management.elasticsearch.ssl.truststore.path: /path/to/file
#xpack.management.elasticsearch.ssl.truststore.password: password

22
docs/static/settings/configuration-management-settings.asciidoc vendored
View file

@ -52,9 +52,9 @@ section in your Logstash configuration, or a different one. Defaults to
If your {es} cluster is protected with basic authentication, these settings
provide the username and password that the Logstash instance uses to
authenticate for accessing the configuration data. The username you specify here
should have the `logstash_admin` role, which provides access to `.logstash-*`
indices for managing configurations.
authenticate for accessing the configuration data.
The username you specify here should have the `logstash_admin` role, which
provides access to `.logstash-*` indices for managing configurations.
`xpack.management.elasticsearch.ssl.certificate_authority`::
@ -78,3 +78,19 @@ the client’s certificate.
`xpack.management.elasticsearch.ssl.keystore.password`::
Optional setting that provides the password to the keystore.
`xpack.management.elasticsearch.cloud_id`::
If you're using {es} in {ecloud}, you should specify the identifier here.
This setting is an alternative to `xpack.management.elasticsearch.hosts`.
If `cloud_id` is configured, `xpack.management.elasticsearch.hosts` should not be used.
This {es} instance will store the Logstash pipeline configurations and metadata.
`xpack.management.elasticsearch.cloud_auth`::
If you're using {es} in {ecloud}, you can set your auth credentials here.
This setting is an alternative to both `xpack.management.elasticsearch.username`
and `xpack.management.elasticsearch.password`. If `cloud_auth` is configured,
those settings should not be used.
The credentials you specify here should be for a user with the `logstash_admin` role, which
provides access to `.logstash-*` indices for managing configurations.

19
docs/static/settings/monitoring-settings.asciidoc vendored
View file

@ -80,3 +80,22 @@ the client’s certificate.
`xpack.monitoring.elasticsearch.ssl.keystore.password`::
Optional settings that provide the password to the keystore.
[[monitoring-additional-settings]]
===== Additional settings
`xpack.monitoring.elasticsearch.cloud_id`::
If you're using {es} in {ecloud}, you should specify the identifier here.
This setting is an alternative to `xpack.monitoring.elasticsearch.hosts`.
If `cloud_id` is configured, `xpack.monitoring.elasticsearch.hosts` should not be used.
The {es} instances that you want to ship your Logstash metrics to. This might be
the same {es} instance specified in the `outputs` section in your Logstash
configuration, or a different one.
`xpack.monitoring.elasticsearch.cloud_auth`::
If you're using {es} in {ecloud}, you can set your auth credentials here.
This setting is an alternative to both `xpack.monitoring.elasticsearch.username`
and `xpack.monitoring.elasticsearch.password`. If `cloud_auth` is configured,
those settings should not be used.

13
x-pack/lib/config_management/elasticsearch_source.rb
View file

@ -35,8 +35,17 @@ module LogStash
def initialize(settings)
super(settings)
if @settings.get("xpack.management.enabled") && !@settings.get_setting("xpack.management.elasticsearch.password").set?
raise ArgumentError.new("You must set the password using the \"xpack.management.elasticsearch.password\" in logstash.yml")
if @settings.get("xpack.management.enabled")
if @settings.get_setting("xpack.management.elasticsearch.cloud_id").set?
if !@settings.get_setting("xpack.management.elasticsearch.cloud_auth").set?
raise ArgumentError.new("You must set credentials using \"xpack.management.elasticsearch.cloud_auth\", " +
"when using \"xpack.management.elasticsearch.cloud_id\" in logstash.yml")
end
else
if !@settings.get_setting("xpack.management.elasticsearch.password").set?
raise ArgumentError.new("You must set the password using \"xpack.management.elasticsearch.password\" in logstash.yml")
end
end
end
@es_options = es_options_from_settings('management', settings)

2
x-pack/lib/config_management/extension.rb
View file

@ -27,6 +27,8 @@ module LogStash
settings.register(LogStash::Setting::NullableString.new("xpack.management.elasticsearch.username", "logstash_system"))
settings.register(LogStash::Setting::NullableString.new("xpack.management.elasticsearch.password"))
settings.register(LogStash::Setting::ArrayCoercible.new("xpack.management.elasticsearch.hosts", String, [ "https://localhost:9200" ] ))
settings.register(LogStash::Setting::NullableString.new("xpack.management.elasticsearch.cloud_id"))
settings.register(LogStash::Setting::NullableString.new("xpack.management.elasticsearch.cloud_auth"))
settings.register(LogStash::Setting::NullableString.new("xpack.management.elasticsearch.ssl.certificate_authority"))
settings.register(LogStash::Setting::NullableString.new("xpack.management.elasticsearch.ssl.truststore.path"))
settings.register(LogStash::Setting::NullableString.new("xpack.management.elasticsearch.ssl.truststore.password"))

52
x-pack/lib/helpers/elasticsearch_options.rb
View file

@ -6,7 +6,16 @@ module LogStash module Helpers
module ElasticsearchOptions
extend self
ES_SETTINGS =%w(ssl.certificate_authority ssl.truststore.path ssl.keystore.path hosts username password)
ES_SETTINGS =%w(
ssl.certificate_authority
ssl.truststore.path
ssl.keystore.path
hosts
username
password
cloud_id
cloud_auth
)
# Retrieve elasticsearch options from either specific settings, or modules if the setting is not there and the
# feature supports falling back to modules if the feature is not specified in logstash.yml
@ -14,14 +23,24 @@ module LogStash module Helpers
only_modules_configured?(feature, settings) ? es_options_from_modules(settings) : es_options_from_settings(feature, settings)
end
# Populate the Elasticsearch options from LogStashSettings file, based on the feature that is being
# used.
# Populate the Elasticsearch options from LogStashSettings file, based on the feature that is being used.
# @return Hash
def es_options_from_settings(feature, settings)
opts = {}
opts['hosts'] = settings.get("xpack.#{feature}.elasticsearch.hosts")
opts['user'] = settings.get("xpack.#{feature}.elasticsearch.username")
opts['password'] = settings.get("xpack.#{feature}.elasticsearch.password")
if cloud_id = settings.get("xpack.#{feature}.elasticsearch.cloud_id")
opts['cloud_id'] = cloud_id
check_cloud_id_configuration!(feature, settings)
else
opts['hosts'] = settings.get("xpack.#{feature}.elasticsearch.hosts")
end
if cloud_auth = settings.get("xpack.#{feature}.elasticsearch.cloud_auth")
opts['cloud_auth'] = cloud_auth
check_cloud_auth_configuration!(feature, settings)
else
opts['user'] = settings.get("xpack.#{feature}.elasticsearch.username")
opts['password'] = settings.get("xpack.#{feature}.elasticsearch.password")
end
opts['sniffing'] = settings.get("xpack.#{feature}.elasticsearch.sniffing")
opts['ssl_certificate_verification'] = settings.get("xpack.#{feature}.elasticsearch.ssl.verification_mode") == 'certificate'
@ -82,7 +101,7 @@ module LogStash module Helpers
modules_configured?(settings) && !feature_configured?(feature, settings)
end
# If not settings are configured, then assume that the feature has not been configured.
# If no settings are configured, then assume that the feature has not been configured.
# The assumption is that with security setup, at least one setting (password or certificates)
# should be configured. If security is not setup, and defaults 'just work' for monitoring, then
# this will need to be reconsidered.
@ -113,4 +132,23 @@ module LogStash module Helpers
# As only one module is supported in the initial rollout, use the first one found
modules_array.first
end
private
def check_cloud_id_configuration!(feature, settings)
return if !settings.set?("xpack.#{feature}.elasticsearch.hosts")
raise ArgumentError.new("Both \"xpack.#{feature}.elasticsearch.cloud_id\" and " +
"\"xpack.#{feature}.elasticsearch.hosts\" specified, please only use one of those.")
end
def check_cloud_auth_configuration!(feature, settings)
return if !settings.set?("xpack.#{feature}.elasticsearch.username") &&
!settings.set?("xpack.#{feature}.elasticsearch.password")
raise ArgumentError.new("Both \"xpack.#{feature}.elasticsearch.cloud_auth\" and " +
"\"xpack.#{feature}.elasticsearch.username\"/\"xpack.#{feature}.elasticsearch.password\" " +
"specified, please only use one of those.")
end
end end end

19
x-pack/lib/monitoring/monitoring.rb
View file

@ -6,12 +6,13 @@ require "logstash/agent"
require "monitoring/internal_pipeline_source"
require "logstash/config/pipeline_config"
require 'helpers/elasticsearch_options'
java_import java.util.concurrent.TimeUnit
module LogStash
class MonitoringExtension < LogStash::UniversalPlugin
include LogStash::Util::Loggable
java_import java.util.concurrent.TimeUnit
class TemplateData
def initialize(node_uuid,
system_api_version,
@ -29,6 +30,8 @@ module LogStash
@es_hosts = es_settings['hosts']
@user = es_settings['user']
@password = es_settings['password']
@cloud_id = es_settings['cloud_id']
@cloud_auth = es_settings['cloud_auth']
@ca_path = es_settings['cacert']
@truststore_path = es_settings['truststore']
@truststore_password = es_settings['truststore_password']
@ -38,7 +41,7 @@ module LogStash
@ssl_certificate_verification = (es_settings['verification_mode'] == 'certificate')
end
attr_accessor :system_api_version, :es_hosts, :user, :password, :node_uuid
attr_accessor :system_api_version, :es_hosts, :user, :password, :node_uuid, :cloud_id, :cloud_auth
attr_accessor :ca_path, :truststore_path, :truststore_password
attr_accessor :keystore_path, :keystore_password, :sniffing, :ssl_certificate_verification
@ -50,6 +53,14 @@ module LogStash
TimeUnit::SECONDS.convert(@collection_timeout_interval, TimeUnit::NANOSECONDS)
end
def cloud_id?
!!cloud_id
end
def cloud_auth?
!!cloud_auth && cloud_id?
end
def auth?
user && password
end
@ -109,7 +120,7 @@ module LogStash
def monitoring_enabled?(settings)
return settings.get_value("xpack.monitoring.enabled") if settings.set?("xpack.monitoring.enabled")
if settings.set?("xpack.monitoring.elasticsearch.hosts")
if settings.set?("xpack.monitoring.elasticsearch.hosts") || settings.set?("xpack.monitoring.elasticsearch.cloud_id")
logger.warn("xpack.monitoring.enabled has not been defined, but found elasticsearch configuration. Please explicitly set `xpack.monitoring.enabled: true` in logstash.yml")
true
else
@ -174,6 +185,8 @@ module LogStash
settings.register(LogStash::Setting::TimeValue.new("xpack.monitoring.collection.timeout_interval", "10m"))
settings.register(LogStash::Setting::NullableString.new("xpack.monitoring.elasticsearch.username", "logstash_system"))
settings.register(LogStash::Setting::NullableString.new("xpack.monitoring.elasticsearch.password"))
settings.register(LogStash::Setting::NullableString.new("xpack.monitoring.elasticsearch.cloud_id"))
settings.register(LogStash::Setting::NullableString.new("xpack.monitoring.elasticsearch.cloud_auth"))
settings.register(LogStash::Setting::NullableString.new("xpack.monitoring.elasticsearch.ssl.certificate_authority"))
settings.register(LogStash::Setting::NullableString.new("xpack.monitoring.elasticsearch.ssl.truststore.path"))
settings.register(LogStash::Setting::NullableString.new("xpack.monitoring.elasticsearch.ssl.truststore.password"))

9
x-pack/lib/template.cfg.erb
View file

@ -12,13 +12,20 @@ input {
}
output {
elasticsearch {
<% if cloud_id? %>
cloud_id => "<%= cloud_id %>"
<% if cloud_auth %>
cloud_auth => "<%= cloud_auth %>"
<% end %>
<% else %>
hosts => <%= es_hosts %>
<% end %>
bulk_path => "/_monitoring/bulk?system_id=logstash&system_api_version=<%= system_api_version %>&interval=1s"
manage_template => false
document_type => "%{[@metadata][document_type]}"
index => ""
sniffing => <%= sniffing %>
<% if auth? %>
<% if auth? && !cloud_auth? %>
user => "<%= user %>"
password => "<%= password %>"
<% end %>

37
x-pack/spec/config_management/elasticsearch_source_spec.rb
View file

@ -103,7 +103,9 @@ describe LogStash::ConfigManagement::ElasticsearchSource do
describe ".new" do
before do
allow_any_instance_of(described_class).to receive(:setup_license_checker)
allow_any_instance_of(described_class).to receive(:license_check)
end
context "when password isn't set" do
let(:settings) do
{
@ -114,10 +116,45 @@ describe LogStash::ConfigManagement::ElasticsearchSource do
#"xpack.management.elasticsearch.password" => elasticsearch_password,
}
end
it "should raise an ArgumentError" do
expect { described_class.new(system_settings) }.to raise_error(ArgumentError)
end
end
context "cloud settings" do
let(:cloud_name) { 'abcdefghijklmnopqrstuvxyz' }
let(:cloud_domain) { 'elastic.co' }
let(:cloud_id) { "label:#{Base64.urlsafe_encode64("#{cloud_domain}$#{cloud_name}$ignored")}" }
let(:settings) do
{
"xpack.management.enabled" => true,
"xpack.management.pipeline.id" => "main",
"xpack.management.elasticsearch.cloud_id" => cloud_id,
"xpack.management.elasticsearch.cloud_auth" => "#{elasticsearch_username}:#{elasticsearch_password}"
}
end
it "should not raise an ArgumentError" do
expect { described_class.new(system_settings) }.not_to raise_error
end
context "when cloud_auth isn't set" do
let(:settings) do
{
"xpack.management.enabled" => true,
"xpack.management.pipeline.id" => "main",
"xpack.management.elasticsearch.cloud_id" => cloud_id,
#"xpack.management.elasticsearch.cloud_auth" => "#{elasticsearch_username}:#{elasticsearch_password}"
}
end
it "should raise an ArgumentError" do
expect { described_class.new(system_settings) }.to raise_error(ArgumentError)
end
end
end
end
describe "#config_path" do

10
x-pack/spec/config_management/extension_spec.rb
View file

@ -41,6 +41,16 @@ describe LogStash::ConfigManagement::Extension do
"xpack.management.elasticsearch.ssl.keystore.password" => [LogStash::Setting::NullableString, nil]
)
it "has a cloud_id setting" do
name = "xpack.management.elasticsearch.cloud_id"
expect { settings.get_setting(name) }.not_to raise_error
end
it "has a cloud_auth setting" do
name = "xpack.management.elasticsearch.cloud_auth"
expect { settings.get_setting(name) }.not_to raise_error
end
describe 'deprecated and renamed settings' do
define_deprecated_and_renamed_settings(
"xpack.management.elasticsearch.url" => "xpack.management.elasticsearch.hosts",

52
x-pack/spec/helpers/elasticsearch_options_spec.rb
View file

@ -15,7 +15,6 @@ shared_examples "elasticsearch options hash is populated without security" do
"user" => expected_username,
"password" => expected_password
)
end
end
@ -109,6 +108,57 @@ describe LogStash::Helpers::ElasticsearchOptions do
it_behaves_like 'elasticsearch options hash is populated without security'
it_behaves_like 'elasticsearch options hash is populated with secure options'
context 'when cloud id and auth are set' do
let(:cloud_name) { 'thebigone'}
let(:cloud_domain) { 'elastic.co'}
let(:cloud_id) { "monitoring:#{Base64.urlsafe_encode64("#{cloud_domain}$#{cloud_name}$ignored")}" }
let(:cloud_username) { 'elastic' }
let(:cloud_password) { 'passw0rd'}
let(:cloud_auth) { "#{cloud_username}:#{cloud_password}" }
let(:expected_url) { ["https://#{cloud_name}.#{cloud_domain}:443"] }
let(:settings) do
{
"xpack.monitoring.enabled" => true,
"xpack.monitoring.elasticsearch.cloud_id" => cloud_id,
"xpack.monitoring.elasticsearch.cloud_auth" => cloud_auth,
}
end
it "creates the elasticsearch output options hash" do
es_options = test_class.es_options_from_settings_or_modules('monitoring', system_settings)
expect(es_options).to include("cloud_id" => cloud_id, "cloud_auth" => cloud_auth)
expect(es_options.keys).to_not include("hosts")
expect(es_options.keys).to_not include("username")
expect(es_options.keys).to_not include("password")
end
context 'hosts also set' do
let(:settings) do
super.merge(
"xpack.monitoring.elasticsearch.hosts" => 'https://localhost:9200'
)
end
it "raises due invalid configuration" do
expect { test_class.es_options_from_settings_or_modules('monitoring', system_settings) }.
to raise_error(ArgumentError, /Both.*?cloud_id.*?and.*?hosts.*?specified/)
end
end
context 'username also set' do
let(:settings) do
super.merge(
"xpack.monitoring.elasticsearch.username" => 'elastic'
)
end
it "raises due invalid configuration" do
expect { test_class.es_options_from_settings_or_modules('monitoring', system_settings) }.
to raise_error(ArgumentError, /Both.*?cloud_auth.*?and.*?username.*?specified/)
end
end
end
end
describe 'es_options_from_settings_or_modules' do