kibana/docs/maps/asset-tracking-tutorial.asciidoc

[role="xpack"]
[[asset-tracking-tutorial]]
== Track, visualize, and alert on assets in real time

Are you interested in asset tracking? Good news! Visualizing and analyzing data that moves is easy with *Maps*. You can track the location of an IoT device and monitor a package or vehicle in transit.

In this tutorial, you’ll look at live urban transit data from the city of Portland, Oregon. You’ll watch the city buses, use the data to visualize congestion, and notify a dispatch team when a bus enters a construction zone.

You’ll learn to:

- Use {agent} to ingest the TriMet REST API into {es}.
- Create a map with layers that visualize asset tracks and last-known locations.
- Use symbols and colors to style data values and show which direction an asset is heading.
- Set up tracking containment alerts to monitor moving vehicles.

When you complete this tutorial, you’ll have a map that looks like this:

[role="screenshot"]
image::maps/images/asset-tracking-tutorial/construction_zones.png[]

[float]
=== Prerequisites

- If you don’t already have {kib}, set it up with https://www.elastic.co/cloud/elasticsearch-service/signup?baymax=docs-body&elektra=docs[our free trial]. Download the deployment credentials.
- Obtain an API key for https://developer.trimet.org/[TriMet web services] at https://developer.trimet.org/appid/registration/.
- {fleet-guide}/fleet-overview.html[Fleet] is enabled on your cluster, and one or more {fleet-guide}/elastic-agent-installation.html[{agent}s] is enrolled.

[float]
=== Part 1: Ingest the Portland bus data
To get to the fun of visualizing and alerting on Portland buses, you must first add the *Custom API* integration to an Elastic Agent policy to get the TriMet Portland bus data into {es}.

[float]
==== Step 1: Set up an Elasticsearch index

. In Kibana, open the main menu, then click *Dev Tools*.
. In *Console*, create the `tri_met_tracks` index lifecyle policy. This policy will keep the events in the hot data phase for 7 days. The data then moves to the warm phase. After 365 days in the warm phase, the data is deleted. 
+
[source,js]
----------------------------------
PUT _ilm/policy/tri_met_tracks
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "50gb",
            "max_age": "7d"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "0d",
        "actions": {
          "set_priority": {
            "priority": 50
          }
        }
      },
      "delete": {
        "min_age": "365d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
----------------------------------
. In *Console*, create the `tri_met_tracks` index template, which is configured to use datastreams:
+
[source,js]
----------------------------------
PUT _index_template/tri_met_tracks
{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "tri_met_tracks"
        }
      }
    },
    "mappings": {
      "_routing": {
        "required": false
      },
      "numeric_detection": false,
      "dynamic_date_formats": [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "dynamic": true,
      "_source": {
        "excludes": [],
        "includes": [],
        "enabled": true
      },
      "dynamic_templates": [],
      "date_detection": true,
      "properties": {
        "trimet": {
          "type": "object",
          "properties": {
            "expires": {
              "type": "date"
            },
            "signMessage": {
              "type": "text"
            },
            "serviceDate": {
              "type": "date"
            },
            "loadPercentage": {
              "type": "float"
            },
            "nextStopSeq": {
              "type": "integer"
            },
            "source": {
              "type": "keyword"
            },
            "type": {
              "type": "keyword"
            },
            "blockID": {
              "type": "integer"
            },
            "signMessageLong": {
              "type": "text"
            },
            "lastLocID": {
              "type": "keyword"
            },
            "nextLocID": {
              "type": "keyword"
            },
            "locationInScheduleDay": {
              "type": "integer"
            },
            "newTrip": {
              "type": "boolean"
            },
            "direction": {
              "type": "integer"
            },
            "inCongestion": {
              "type": "boolean"
            },
            "routeNumber": {
              "type": "integer"
            },
            "bearing": {
              "type": "integer"
            },
            "garage": {
              "type": "keyword"
            },
            "tripID": {
              "type": "keyword"
            },
            "delay": {
              "type": "integer"
            },
            "extraBlockID": {
              "type": "keyword"
            },
            "messageCode": {
              "type": "integer"
            },
            "lastStopSeq": {
              "type": "integer"
            },
            "location": {
              "type": "geo_point"
            },
            "time": {
              "index": true,
              "ignore_malformed": false,
              "store": false,
              "type": "date",
              "doc_values": true
            },
            "vehicleID": {
              "type": "keyword"
            },
            "offRoute": {
              "type": "boolean"
            }
          }
        }
      }
    }
  },
  "index_patterns": [
    "tri_met_tracks*"
  ],
  "data_stream": {
    "hidden": false,
    "allow_custom_routing": false
  },
  "composed_of": []
}
----------------------------------
. In **Console**, add the `tri_met_track` ingest pipeline. 
+
[source,js]
----------------------------------
PUT _ingest/pipeline/tri_met_tracks
{
  "processors": [
    {
      "set": {
        "field": "trimet.inCongestion",
        "value": "false",
        "if": "ctx?.trimet?.inCongestion == null"
      }
    },
    {
      "convert": {
        "field": "trimet.bearing",
        "type": "float"
      }
    },
    {
      "convert": {
        "field": "trimet.inCongestion",
        "type": "boolean"
      }
    },
    {
      "script": {
        "source": "ctx['trimet']['location'] = ctx['trimet']['latitude'] + \",\" + ctx['trimet']['longitude']"
      }
    },
    {
      "script": {
        "source": "ctx['_id'] = ctx['trimet']['vehicleID'] + \"_\" + ctx['trimet']['time']",
        "description": "Generate documentID"
      }
    },
    {
      "remove": {
        "field": [
          "message",
          "input",
          "agent",
          "ecs",
          "host",
          "event",
          "trimet.longitude",
          "trimet.latitude"
        ]
      }
    },
    {
      "set": {
        "field": "_index",
        "value": "tri_met_tracks"
      }
    }
  ]
}
----------------------------------

[float]
==== Step 2: Configure {agent}

. From the {kib} main menu, click *Fleet*, then the *Agent policies* tab.

. Click the name of the agent policy where you want to add the *Custom API* integration. The configuration changes you make only apply to the policy you select.

. Click the name of the *Custom API* integration, or add the integration if the agent policy does not yet have it.

. From the *Edit Custom API integration* page, expand the *Change defaults* section.

. Set the *Dataset name* to *httpjson.trimet*.

. Set the *Ingest Pipeline* to *tri_met_pipeline*.

. Set the *Request URL* to *https://developer.trimet.org/ws/v2/vehicles?appID=<tri_met_app_id>*.

. Set *Response Split* to *target: body.resultSet.vehicle*.

. At the bottom of the configuration, expand *Advanced options*.

. Set *Processors* to:
+
[source,yaml]
----------------------------------
- decode_json_fields:
    fields: ["message"]
    target: "trimet"
----------------------------------

. Leave everything else as defaults.

. Click *Save integration* to deploy the configuration to any {agent} with the policy assigned.


[float]
==== Step 3: Create a data view for the tri_met_tracks {es} index

. In {kib}, open the main menu, and click *Stack Management > Data Views*.
. Click *Create data view*.
. Give the data view a name: *tri_met_tracks**.
. Set the index pattern as: *tri_met_tracks**.
. Set the *Timestamp field* to *trimet.time*.
. Click *Save data view to Kibana*.

{kib} shows the fields in your data view.

[role="screenshot"]
image::maps/images/asset-tracking-tutorial/data_view.png[]

[float]
==== Step 4: Explore the Portland bus data

. Open the main menu, and click *Discover*.
. Set the data view to *tri_met_tracks**.
. Open the <<set-time-filter, time filter>>, and set the time range to the last 15 minutes.
. Expand a document and explore some of the fields that you will use later in this tutorial: `trimet.bearing`, `trimet.inCongestion`, `trimet.location`, and `trimet.vehicleID`.

[role="screenshot"]
image::maps/images/asset-tracking-tutorial/discover.png[]

[float]
=== Part 2: Build an operational map
It's hard to get an overview of Portland buses by looking at individual events. Let's create a map to show the bus routes and current location for each bus, along with the direction the buses are heading.

[float]
==== Step 1: Create your map
Create your map and set the theme for the default layer to dark mode.

. Open the main menu, and click *Maps*.
. Click *Create map*.
. In the *Layers* list, click *Road map*, and then click *Edit layer settings*.
. Open the *Tile service* dropdown, and select *Road map - dark*.
. Click *Keep changes*.

[float]
==== Step 2. Add a tracks layer

Add a layer to show the bus routes for the last 15 minutes.

. Click *Add layer*.
. Click *Tracks*.
. Select the *tri_met_tracks** data view.
. Define the tracks:
.. Set *Entity* to *trimet.vehicleID*.
.. Set *Sort* to *trimet.time*.
. Click *Add and continue*.
. In Layer settings:
.. Set *Name* to *Buses*.
.. Set *Opacity* to 80%.
. Scroll to *Layer Style*, and set *Border color* to pink.
. Click *Keep changes*.
. In the *Layers* list, click *Buses*, and then click *Fit to data*.

At this point, you have a map with lines that represent the routes of the buses as they move around the city.

[role="screenshot"]
image::maps/images/asset-tracking-tutorial/tracks_layer.png[]

[float]
==== Step 3. Indicate the direction of the bus tracks

Add a layer that uses attributes in the data to set the style and orientation of the buses. You’ll see the direction buses are headed and what traffic is like.

. Click *Add layer*, and then select *Top Hits per entity*.
. Select the *tri_met_tracks** data view.
. To display the most recent location per bus:
.. Set *Entity* to *trimet.vehicleID*.
.. Set *Documents per entity* to 1.
.. Set *Sort field* to *trimet.time*.
.. Set *Sort order* to *descending*.
. Click *Add and continue*.
. Scroll to *Layer Style*.
.. Set *Symbol type* to *icon*.
.. Set *Icon* to *arrow-es*.
.. Set the *Fill color*:
... Select *By value* styling, and set the field to *trimet.inCongestion*.
... Use a *Custom color palette*.
... Set the *Other* color to black.
... Add a green class for *false*, meaning the bus is not in traffic.
... Add a red class for *true*, meaning the bus is in congestion.
.. Set *Border width* to 0.
.. Change *Symbol orientation* to use *By value* and the *trimet.bearing* field.
+
[role="screenshot"]
image::maps/images/asset-tracking-tutorial/top_hits_layer_style.png[]
. Click *Keep changes*.
. Open the <<set-time-filter, time filter>>, and set *Refresh every* to 10 seconds, and click *Start*.

Your map should automatically refresh every 10 seconds to show the latest bus positions and tracks.

[role="screenshot"]
image::maps/images/asset-tracking-tutorial/tracks_and_top_hits.png[]

[float]
=== Part 3: Setup geo-fencing alerts
Let's make TriMet Portland bus data actionable and alert when buses enter construction zones.

[float]
==== Step 1. Add a construction zone

Add a layer for construction zones, which you will draw on the map. The construction zones will be used as your geofence boundary or threshold that serves as the basis for triggering alerts.

. Click *Add layer*.
. Click *Create index*.
. Set *Index name* to *trimet_construction_zones*.
. Click *Create index*.
. Draw 2 or 3 construction zones on your map:
.. In the toolbar on left side of the map, select the bounding box icon image:maps/images/asset-tracking-tutorial/bounding_box_icon.png[bounding box icon].
.. To draw a construction zone, click a start point on the map and drag.
.. Click an endpoint to finish.
. When you finish drawing the construction zones, click *Exit* under the layer name in the legend.
. In *Layer settings*, set *Name* to *Construction zones*.
. Scroll to *Layer Style*, and set *Fill color* to yellow.
. Click *Keep changes*.
. *Save* the map.
.. Give the map a title.
.. Under *Add to dashboard*, select *None*.
.. Click *Save and add to library*.

The map now represents an operational view of live bus traffic.  You’ll see the direction that the buses are traveling, and whether they are near or have entered a construction zone.

Your map is now complete.

[role="screenshot"]
image::maps/images/asset-tracking-tutorial/construction_zones.png[]


[float]
==== Step 2. Configure an alert

Create a new alert by defining a rule and a connector. The rule includes the conditions that will trigger the alert, and the connector defines what action takes place once the alert is triggered. In this case, each alert will log a message to the {kib} log. 

NOTE: For this example, you will set the rule to check every 5 seconds. However, when running in production, consider setting a higher check interval (such as 1 minute) to avoid performance issues. Refer to <<alerting-production-considerations,Alerting production considerations>> for more information.

. Open *{stack-manage-app}*, and then click *{rules-ui}*.
. Click *Create rule*.
. Name the rule *Bus Alerts*.
. Set *Check every* to *5 seconds*.
. Notify *Only on status change*.
+
[role="screenshot"]
image::maps/images/asset-tracking-tutorial/rule_configuration.png[]
. Select the *Tracking containment* rule type.
. Set *Select entity*:
.. Set *INDEX* to *tri_met_tracks**.
.. Set *BY* to *trimet.vehicleID*.
. Set *Select boundary* *INDEX* to *trimet_construction_zones*.
+
[role="screenshot"]
image::maps/images/asset-tracking-tutorial/tracking_containment_configuration.png[]
. Under *Actions*, select the *Server log* connector type.
. Click *Create a connector*.
. In the *Server log connector*:
.. Set *Connector name* to *Bus alert connector*.
.. Click *Save*.
. Complete the *Actions* configuration.
.. Set *Message* to :
+
[source,js]
----------------------------------
{
  "entityId": "{{context.entityId}}",
  "entityDateTime": "{{context.entityDateTime}}",
  "entityDocumentId": "{{context.entityDocumentId}}",
  "detectionDateTime": "{{context.detectionDateTime}}",
  "entityLocation": "{{context.entityLocation}}",
  "containingBoundaryId": "{{context.containingBoundaryId}}",
  "containingBoundaryName": "{{context.containingBoundaryName}}"
}
----------------------------------

. Click *Save*.

The *Bus Alert connector* is added to the *{connectors-ui}* page. For more information on common connectors, refer to the <<slack-action-type, Slack>> and <<email-action-type,Email>> connectors.

Congratulations! You have completed the tutorial and have the recipe for tracking assets. You can now try replicating this same analysis with your own data.