[[security-basic-setup-https]] === Set up basic security for the Elastic Stack plus secured HTTPS traffic ++++ Set up basic security plus HTTPS ++++ When you enable TLS on the HTTP layer it provides an additional layer of security to ensure that all communications to and from your cluster are encrypted. When you run the `elasticsearch-certutil` tool in `http` mode, the tool asks several questions about how you want to generate certificates. While there are numerous options, the following choices result in certificates that should work for most environments. [[signing-certificates]] .Signing certificates **** The first question that the `elasticsearch-certutil` tool prompts you with is whether you want to generate a Certificate Signing Request (CSR). Answer `n` if you want to sign your own certificates, or `y` if you want to sign certificates with a central CA. [discrete] ===== Sign your own certificates If you want to use the CA that you created when <> answer `n` when asked if you want to generate a CSR. You then specify the location of your CA, which the tool uses to sign and generate a `.p12` certificate. The steps in this procedure follow this workflow. [discrete] ===== Sign certificates with a central CA If you work in an environment with a central security team, they can likely generate a certificate for you. Infrastructure within your organization might already be configured to trust an existing CA, so it may be easier for clients to connect to {es} if you use a CSR and send that request to the team that controls your CA. To use a central CA, answer `y` to the first question. **** [[basic-setup-https-prerequisites]] ==== Prerequisites Complete all steps in <>. [[encrypt-http-communication]] ==== Encrypt HTTP client communications for {es} . On *every* node in your cluster, stop {es} and {kib} if they are running. . On any single node, from the directory where you installed {es}, run the {es} HTTP certificate tool to generate a Certificate Signing Request (CSR). + [source,shell] ---- ./bin/elasticsearch-certutil http ---- + This command generates a `.zip` file that contains certificates and keys to use with {es} and {kib}. Each folder contains a `README.txt` explaining how to use these files. a. When asked if you want to generate a CSR, enter `n`. b. When asked if you want to use an existing CA, enter `y`. c. Enter the path to your CA. This is the absolute path to the `elastic-stack-ca.p12` file that you generated for your cluster. d. Enter the password for your CA. e. Enter an expiration value for your certificate. You can enter the validity period in years, months, or days. For example, enter `90D` for 90 days. f. When asked if you want to generate one certificate per node, enter `y`. + Each certificate will have its own private key, and will be issued for a specific hostname or IP address. g. When prompted, enter the name of the first node in your cluster. Use the same node name that you used when <>. h. Enter all hostnames used to connect to your first node. These hostnames will be added as DNS names in the Subject Alternative Name (SAN) field in your certificate. + List every hostname and variant used to connect to your cluster over HTTPS. i. Enter the IP addresses that clients can use to connect to your node. j. Repeat these steps for each additional node in your cluster. . After generating a certificate for each of your nodes, enter a password for your private key when prompted. . Unzip the generated `elasticsearch-ssl-http.zip` file. This compressed file contains one directory for both {es} and {kib}. + -- [source,txt] ---- /elasticsearch |_ README.txt |_ http.p12 |_ sample-elasticsearch.yml ---- [source,txt] ---- /kibana |_ README.txt |_ elasticsearch-ca.pem |_ sample-kibana.yml ---- -- . On *every* node in your cluster, complete the following steps: .. Copy the relevant `http.p12` certificate to the `$ES_PATH_CONF` directory. .. Edit the `elasticsearch.yml` file to enable HTTPS security and specify the location of the `http.p12` security certificate. + [source,yaml] ---- xpack.security.http.ssl.enabled: true xpack.security.http.ssl.keystore.path: http.p12 ---- .. Add the password for your private key to the secure settings in {es}. + [source,shell] ---- ./bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password ---- .. Start {es}. **Next**: <> [[encrypt-kibana-http]] ==== Encrypt HTTP client communications for {kib} Browsers send traffic to {kib} and {kib} sends traffic to {es}. These communication channels are configured separately to use TLS. You encrypt traffic between {kib} and {es}, and then encrypt traffic between your browser and {kib}. [[encrypt-kibana-elasticsearch]] ===== Encrypt traffic between {kib} and {es} When you ran the `elasticsearch-certutil` tool with the `http` option, it created a `/kibana` directory containing an `elasticsearch-ca.pem` file. You use this file to configure {kib} to trust the {es} CA for the HTTP layer. . Copy the `elasticsearch-ca.pem` file to the {kib} configuration directory, as defined by the `$KBN_PATH_CONF` path. . Open `kibana.yml` and add the following line to specify the location of the security certificate for the HTTP layer. + [source,yaml] ---- elasticsearch.ssl.certificateAuthorities: $KBN_PATH_CONF/elasticsearch-ca.pem ---- . Add the following line to specify the HTTPS URL for your {es} cluster. + [source,yaml] ---- elasticsearch.hosts: https://:9200 ---- . Restart {kib}. .Connect to a secure monitoring cluster **** If the Elastic monitoring features are enabled and you configured a separate {es} monitoring cluster, you can also configure {kib} to connect to the monitoring cluster via HTTPS. The steps are the same, but each setting is prefixed by `monitoring`. For example, `monitoring.ui.elasticsearch.hosts` and `monitoring.ui.elasticsearch.ssl.truststore.path`. NOTE: You must create a separate `elasticsearch-ca.pem` security file for the monitoring cluster. **** **Next**: <> [[encrypt-kibana-browser]] ===== Encrypt traffic between your browser and {kib} You create a server certificate and private key for {kib}. {kib} uses this server certificate and corresponding private key when receiving connections from web browsers. When you obtain a server certificate, you must set its subject alternative name (SAN) correctly to ensure that browsers will trust it. You can set one or more SANs to the {kib} server’s fully-qualified domain name (FQDN), hostname, or IP address. When choosing the SAN, pick whichever attribute you'll use to connect to {kib} in your browser, which is likely the FQDN. The following instructions create a Certificate Signing Request (CSR) for {kib}. A CSR contains information that a CA uses to generate and sign a security certificate. The certificate can be trusted (signed by a public, trusted CA) or untrusted (signed by an internal CA). A self-signed or internally-signed certificate is acceptable for development environments and building a proof of concept, but should not be used in a production environment. WARNING: Before going to production, use a trusted CA such as https://letsencrypt.org/[Let's Encrypt] or your organization's internal CA to sign the certificate. Using a signed certificate establishes browser trust for connections to {kib} for internal access or on the public internet. . Generate a server certificate and private key for {kib}. + [source,shell] ---- ./bin/elasticsearch-certutil csr -name kibana-server -dns example.com,www.example.com ---- + The CSR has a common name (CN) of `kibana-server`, a SAN of `example.com`, and another SAN of `www.example.com`. + This command generates a `csr-bundle.zip` file by default with the following contents: + [source,txt] ---- /kibana-server |_ kibana-server.csr |_ kibana-server.key ---- . Unzip the `csr-bundle.zip` file to obtain the `kibana-server.csr` unsigned security certificate and the `kibana-server.key` unencrypted private key. . Send the `kibana-server.csr` certificate signing request to your internal CA or trusted CA for signing to obtain a signed certificate. The signed file can be in different formats, such as a `.crt` file like `kibana-server.crt`. . Open `kibana.yml` and add the following lines to configure {kib} to access the server certificate and unencrypted private key. + [source,yaml] ---- server.ssl.certificate: $KBN_PATH_CONF/kibana-server.crt server.ssl.key: $KBN_PATH_CONF/kibana-server.key ---- + NOTE: `$KBN_PATH_CONF` contains the path for the {kib} configuration files. If you installed {kib} using archive distributions (`zip` or `tar.gz`), the path defaults to `$KBN_HOME/config`. If you used package distributions (Debian or RPM), the path defaults to `/etc/kibana`. . Add the following line to `kibana.yml` to enable TLS for inbound connections. + [source,yaml] ---- server.ssl.enabled: true ---- . Start {kib}. NOTE: After making these changes, you must always access {kib} via HTTPS. For example, `https://.com`. **Next**: <> [[configure-beats-security]] ==== Configure {beats} security {beats} are open source data shippers that you install as agents on your servers to send operational data to {es}. Each Beat is a separately installable product. The following steps cover configuring security for {metricbeat}. Follow these steps for each {beats-ref}/getting-started.html[additional Beat] you want to configure security for. ===== Prerequisites {metricbeat-ref}/metricbeat-installation-configuration.html[Install {metricbeat}] using your preferred method. IMPORTANT: You cannot connect to the {stack} or configure assets for {metricbeat} before completing the following steps. ===== Create roles for {metricbeat} Typically, you need to create the following separate roles: - **setup** role for setting up index templates and other dependencies - **monitoring** role for sending monitoring information - **writer** role for publishing events collected by {metricbeat} - **reader** role for Kibana users who need to view and create visualizations that access {metricbeat} data NOTE: These instructions assume that you are using the default name for {metricbeat} indices. If the indicated index names are not listed, or you are using a custom name, enter it manually when defining roles and modify the privileges to match your index naming pattern. To create users and roles from Stack Management in {kib}, select **Roles** or **Users** from the side navigation. **Next**: <> [discrete] [[beats-setup-role]] ====== Create a setup role and user Administrators who set up {metricbeat} typically need to load mappings, dashboards, and other objects used to index data into {es} and visualize it in {kib}. WARNING: Setting up {metricbeat} is an admin-level task that requires extra privileges. As a best practice, grant the setup role to administrators only, and use a more restrictive role for event publishing. . Create the setup role: . Enter **metricbeat_setup** as the role name. . Choose the **monitor** and **manage_ilm** cluster privileges. . On the **metricbeat-\*** indices, choose the **manage** and **write** privileges. + If the **metricbeat-\*** indices aren't listed, enter that pattern into the list of indices. . Create the setup user: . Enter **metricbeat_setup** as the user name. . Enter the username, password, and other user details. . Assign the following roles to the **metricbeat_setup** user: + [cols="1,1"] |=== | Role | Purpose | `metricbeat_setup` | Set up {metricbeat}. | `kibana_admin` | Load dependencies, such as example dashboards, if available, into {kib} | `ingest_admin` | Set up index templates and, if available, ingest pipelines |=== **Next**: <> [discrete] [[beats-monitoring-role]] ====== Create a monitoring role and user To send monitoring data securely, create a monitoring user and grant it the necessary privileges. You can use the built-in `beats_system` user, if it’s available in your environment. Because the built-in users are not available in {ecloud}, these instructions create a user that is explicitly used for monitoring {metricbeat}. . If you're using the built-in `beats_system` user, on any node in your cluster, run the <> utility to set the password for that user: + This command resets the password for the `beats_system` user to an auto-generated value. + [source,shell] ---- ./bin/elasticsearch-reset-password -u beats_system ---- + If you want to set the password to a specific value, run the command with the interactive (`-i`) parameter. + [source,shell] ---- ./bin/elasticsearch-reset-password -i -u beats_system ---- . Create the monitoring role: . Enter **metricbeat_monitoring** as the role name. . Choose the **monitor** cluster privilege. . On the **.monitoring-beats-\*** indices, choose the **create_index** and **create_doc** privileges. . Create the monitoring user: . Enter **metricbeat_monitoring** as the user name. . Enter the username, password, and other user details. . Assign the following roles to the **metricbeat_monitoring** user: + [cols="1,1"] |=== | Role | Purpose | `metricbeat_monitoring` | Monitor {metricbeat}. | `kibana_admin` | Use {kib} | `monitoring_user` | Use Stack Monitoring in {kib} to monitor {metricbeat} |=== **Next**: <> [discrete] [[beats-writer-role]] ====== Create a writer role and user Users who publish events to {es} need to create and write to {metricbeat} indices. To minimize the privileges required by the writer role, use the setup role to pre-load dependencies. This section assumes that you’ve <>. . Create the writer role: . Enter **metricbeat_writer** as the role name. . Choose the **monitor** and **read_ilm** cluster privileges. . On the **metricbeat-\*** indices, choose the **create_doc**, **create_index**, and **view_index_metadata** privileges. . Create the writer user: . Enter **metricbeat_writer** as the user name. . Enter the username, password, and other user details. . Assign the following roles to the **metricbeat_writer** user: + [cols="1,1"] |=== | Role | Purpose | `metricbeat_writer` | Monitor {metricbeat} | `remote_monitoring_collector` | Collect monitoring metrics from {metricbeat} | `remote_monitoring_agent` | Send monitoring data to the monitoring cluster |=== **Next**: <> [discrete] [[beats-reader-role]] ====== Create a reader role and user {kib} users typically need to view dashboards and visualizations that contain {metricbeat} data. These users might also need to create and edit dashboards and visualizations. Create the reader role to assign proper privileges to these users. . Create the reader role: . Enter **metricbeat_reader** as the role name. . On the **metricbeat-\*** indices, choose the **read** privilege. . Under **Kibana**, click **Add Kibana privilege**. - Under **Spaces**, choose **Default**. - Choose **Read** or **All** for Discover, Visualize, Dashboard, and Metrics. . Create the reader user: . Enter **metricbeat_reader** as the user name. . Enter the username, password, and other user details. . Assign the following roles to the **metricbeat_reader** user: + [cols="1,1"] |=== | Role | Purpose | `metricbeat_reader` | Read {metricbeat} data. | `monitoring_user` | Allow users to monitor the health of {metricbeat} itself. Only assign this role to users who manage {metricbeat} | `beats_admin` | Create and manage configurations in {beats} central management. Only assign this role to users who need to use {beats} central management. |=== **Next**: <> [discrete] [[configure-metricbeat-tls]] ===== Configure {metricbeat} to use TLS Before starting {metricbeat}, you configure the connections to {es} and {kib}. You can configure authentication to send data to your secured cluster using basic authentication, API key authentication, or Public Key Infrastructure (PKI) certificates. The following instructions use the credentials for the `metricbeat_writer` and `metricbeat_setup` users that you created. If you need a greater level of security, we recommend using PKI certificates. After configuring connections to {es} and {kib}, you'll enable the `elasticsearch-xpack` module and configure that module to use HTTPS. WARNING: In production environments, we strongly recommend using a separate cluster (referred to as the monitoring cluster) to store your data. Using a separate monitoring cluster prevents production cluster outages from impacting your ability to access your monitoring data. It also prevents monitoring activities from impacting the performance of your production cluster. . On the node where you <>, navigate to the `/kibana` directory. . Copy the `elasticsearch-ca.pem` certificate to the directory where you installed {metricbeat}. . Open the `metricbeat.yml` configuration file and configure the connection to {es}. + Under `output.elasticsearch`, specify the following fields: + [source,yaml] ---- output.elasticsearch: hosts: [":9200"] protocol: "https" username: "metricbeat_writer" password: "" ssl: certificate_authorities: ["elasticsearch-ca.pem"] verification_mode: "certificate" ---- `hosts`:: Specifies the host where your {es} cluster is running. `protocol`:: Indicates the protocol to use when connecting to {es}. This value must be `https`. `username`:: Name of the user with privileges required to publish events to {es}. The `metricbeat_writer` user that you created has these privileges. `password`:: Password for the indicated `username`. `certificate_authorities`:: Indicates the path to the local `.pem` file that contains your CA's certificate. . Configure the connection to {kib}. + Under `setup.kibana`, specify the following fields: + [source,yaml] ---- setup.kibana host: "https://:5601" ssl.enabled: true username: "metricbeat_setup" password: "p@ssw0rd" ---- `hosts`:: The URLs of the {es} instances to use for all your queries. Ensure that you include `https` in the URL. `username`:: Name of the user with privileges required to set up dashboards in {kib}. The `metricbeat_setup` user that you created has these privileges. `password`:: Password for the indicated `username`. . Enable the `elasticsearch-xpack` module. + [source,shell] ---- ./metricbeat modules enable elasticsearch-xpack ---- . Modify the `elasticsearch-xpack` module to use HTTPS. This module collects metrics about {es}. + Open `/modules.d/elasticsearch-xpack.yml` and specify the following fields: + [source,yaml] ---- - module: elasticsearch xpack.enabled: true period: 10s hosts: ["https://:9200"] username: "remote_monitoring_user" password: "" ssl: <1> enabled: true certificate_authorities: ["elasticsearch-ca.pem"] verification_mode: "certificate" ---- <1> Configuring SSL is required when monitoring a node with encrypted traffic. See {metricbeat-ref}/configuration-ssl.html[Configure SSL for {metricbeat}]. `hosts`:: Specifies the host where your {es} cluster is running. Ensure that you include `https` in the URL. `username`:: Name of the user with privileges to collect metric data. The built-in `monitoring_user` user has these privileges. Alternatively, you can create a user and assign it the `monitoring_user` role. `password`:: Password for the indicated `username`. `certificate_authorities`:: Indicates the path to the local `.pem` file that contains your CA's certificate. . If you want to use the predefined assets for parsing, indexing, and visualizing your data, run the following command to load these assets: + [source,shell] ---- ./metricbeat setup -e ---- . Start {es}, and then start {metricbeat}. + [source,shell] ---- ./metricbeat -e ---- + `-e` is optional and sends output to standard error instead of the configured log output. . Log in to {kib}, open the main menu, and click **Stack Monitoring**. + You’ll see cluster alerts that require your attention and a summary of the available monitoring metrics for {es}. Click any of the header links on the available cards to view additional information.