Overview

Sometimes users have issues logging into Atlassian Stash due to a CAPTCHA input screen they can not see when using Google Chrome. When using Microsoft Internet Explorer, Apple Safari, or another web browser they can see the CAPTCHA screen and resolve their issue rather quickly. Otherwise it's very confusing for the user.

The purpose of this plugin is to proactively monitor Atlassian Stash logs and look for users experiencing this problem and notify them before they become very frustrated.

Steps required to accomplish this:

Capture Atlassian Stash auth logs via logstash
Parse Atlassian Stash auth logs with custom grok pattern looking explicitly for CAPTCHA errors
Send Atlassian Stash CAPTCHA events to a custom logstash filter that will use the Atlassian Crowd REST service to map a user email to username
Update Elasticsearch with the email to username mapping
Create Kibana dashboard that shows users having CAPTCHA problem when attempting to login to Atlassian Stash

Capture and Ship auth log

Capture Atlassian Stash auth logs via logstash ship them to redis

The logstash config code snippet below is used on the Stash server.

input {
  file {
    type    => "stash-auth"
    path    => "/stash-home/log/audit/atlassian-stash-audit.*"
    exclude => "*.gz"
    tags    => ["stash","stash-auth"]
  }
}
filter {
 
}
output {
  redis {
    host      => "redis"
    data_type => "list"
    key       => "somevalue"
  }
}

Pull from redis and parse

Pull Atlassian Stash auth events from redis and parse with custom grok filter looking for CAPTCHA errors

The logstash config code snippet below is used on the central logstash indexing server which pulls events from multiple redis queues, and runs multiple instances of logstash.

input {
  redis {
    host        => "redis"
    batch_count => 1024
    threads     => 16
    data_type   => "list"
    key         => "somevalue"
    codec       => json
    type        => "redis"
  }
}
filter {
  if "stash-auth" in [type] {
    grok {
      patterns_dir => "patterns"
      pattern      => "%{STASH_CAPTCHA}"
      add_tag      => ["stash-captcha"]
    }
    if "stash-captcha" in [tags] {
      crowd {
       crowdURL       => "https://crowd/rest/usermanagement/1/user"
       crowdUsername  => "username"
       crowdPassword  => "password"
       timeout        => 2
       username_field => "user1"
      }
    }
  }
}
output {
  elasticsearch_http {
    host => "elasticsearch"
  }
}

Grok pattern

The logstash configuration above uses a custom grok filter called STASH_CAPTCHA. Below is that custom grok filter.

STASH_CAPTCHA %{IP:proxy},%{IP:client} \| %{WORD:error} \| %{WORD:user1} \| %{INT:epoch_time} \| %{WORD:user2} \| (?<error>{%{QS}:%{QS},%{QS}:"For security reasons you must answer a CAPTCHA question."}) \| %{INT:minuteinday}x%{INT:reqnumsincerestart}x%{INT:concurrentreqs} \| %{DATA:something}

Notice how user1 captures the username used in the failed authentication event. Below is an example of a failed authentication event with the CAPTCHA error. Also notice how the actual log message from the Stash auth log does NOT contain the user's email address. Our organization has thousands of Stash users, and it's impossible for the team supporting Stash to know by a username in a log what the user's email address is. It's something that can be looked up in the Crowd or Stash admin web UI, but that takes time and requires each user be looked up individually. The end result of this process will be one Kibana Dashboard with the top CAPTCHA error users who appear to be having the issue multiple times.

10.0.0.1,127.0.0.1 | AuthenticationFailureEvent | MICKEY | 1404773427356 | mickey | {"authentication-method":"basic","error":"For security reasons you must answer a CAPTCHA question."} | 1070x5886652x4 | -

REST Service

Calling Atlassian Crowd REST service

Now that we have filtered the event and have a logstash event containing a user1 field with the username, we can pass the event to another custom logstash filter that looks the username up in the Atlassian Crowd REST service and returns the users email address.

Before doing this the Atlassian Crowd service needs to be configured to allow a new application access to the REST service, and the IP of the logstash server connecting to the REST service needs to be allowed access.

Part 2 of this tutorial which makes the Atlassian Crowd REST service call explains the custom plugin in detail.

Final Dashboard

Once your data is in elasticsearch you can create a custom dashboard. Or even script email notices to users who appear to be having issues.

Hello! I'm your friendly footer. If you're actually reading this, I'm impressed. :)