mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-04-24 23:27:25 -04:00
[DOCS] Remove 'Watching event data' example (#92872)
This commit is contained in:
Abdon Pijpelink
GitHub
parent
85e965a35c
commit
1bb660c810
3 changed files with 12 additions and 316 deletions
|
|
@ -1868,3 +1868,11 @@ For more information about reindexing from a remote cluster, refer to
|
|||
=== Infer trained model deployment API
|
||||
|
||||
See <<infer-trained-model>>.
|
||||
|
||||
[role="exclude",id="watching-meetup-data"]
|
||||
=== Watching event data
|
||||
|
||||
This example has been removed, as the Meetup.com streaming API has been
|
||||
discontinued.
|
||||
|
||||
Refer to <<example-watches>> for other Watcher examples.
|
||||
|
|
@ -1,15 +1,13 @@
|
|||
[role="xpack"]
|
||||
[[example-watches]]
|
||||
== Example watches
|
||||
The following examples show how to set up watches to:
|
||||
The following example shows how to set up a watch to:
|
||||
|
||||
* <<watch-cluster-status,Monitor the status of an Elasticsearch cluster>>
|
||||
* <<watching-meetup-data,Monitor event data>>
|
||||
|
||||
For more example watches you can use as a starting point for building custom
|
||||
watches, see the https://github.com/elastic/examples/tree/master/Alerting[Example
|
||||
watches] in the Elastic Examples repo.
|
||||
watches, refer to the
|
||||
https://github.com/elastic/examples/tree/master/Alerting[Example watches] in the
|
||||
Elastic Examples repo.
|
||||
|
||||
include::example-watches/example-watch-clusterstatus.asciidoc[]
|
||||
|
||||
include::example-watches/example-watch-meetupdata.asciidoc[]
|
||||
|
|
|
|||
|
|
@ -1,310 +0,0 @@
|
|||
[role="xpack"]
|
||||
[[watching-meetup-data]]
|
||||
=== Watching event data
|
||||
|
||||
If you are indexing event data, such as log messages, network traffic, or a web feed, you can create a watch to email notifications when certain events occur.
|
||||
For example, if you index a feed of RSVPs for meetup events happening around the world, you can create a watch that alerts you to interesting events.
|
||||
|
||||
To index the meetup data, you can use https://www.elastic.co/products/logstash[Logstash] to ingest live data from the Meetup.com streaming API, `http://stream.meetup.com/2/rsvps`.
|
||||
|
||||
To ingest this data with Logstash:
|
||||
|
||||
. https://www.elastic.co/downloads/logstash[Download Logstash] and unpack the
|
||||
archive file.
|
||||
|
||||
. Create a Logstash configuration file that uses the {logstash-ref}/plugins-inputs-stdin.html[Logstash standard input] and the {logstash-ref}/plugins-outputs-stdout.html[Logstash standard output] and save it in `logstash-{version}` directory as `livestream.conf`:
|
||||
+
|
||||
--
|
||||
[source,ruby]
|
||||
----------------------------------------------------------
|
||||
input {
|
||||
stdin {
|
||||
codec => json <1>
|
||||
}
|
||||
}
|
||||
filter {
|
||||
date {
|
||||
match => [ "event.time", "UNIX_MS" ]
|
||||
target => "event_time"
|
||||
}
|
||||
}
|
||||
output { <2>
|
||||
stdout {
|
||||
codec => rubydebug
|
||||
}
|
||||
elasticsearch {
|
||||
hosts => "http://localhost:9200"
|
||||
user => "elastic"
|
||||
password => "x-pack-test-password"
|
||||
}
|
||||
}
|
||||
|
||||
----------------------------------------------------------
|
||||
// NOTCONSOLE
|
||||
<1> The meetup data is formatted in JSON.
|
||||
<2> Index the meetup data into Elasticsearch.
|
||||
--
|
||||
|
||||
. To start indexing the meetup data, pipe the RSVP stream into Logstash and specify your `livestream.conf` configuration file.
|
||||
+
|
||||
--
|
||||
[source,shell]
|
||||
----------------------------------------------------------
|
||||
curl http://stream.meetup.com/2/rsvps | bin/logstash -f livestream.conf
|
||||
----------------------------------------------------------
|
||||
// NOTCONSOLE
|
||||
--
|
||||
|
||||
Now that you're indexing the meetup RSVPs, you can set up a watch that lets you know about events you might be interested in. For example, let's create a watch that runs every hour, looks for events that talk about _Open Source_, and sends an email with information about the events.
|
||||
|
||||
|
||||
To set up the watch:
|
||||
|
||||
. Specify how often you want to run the watch by adding a schedule trigger to the watch:
|
||||
+
|
||||
--
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
{
|
||||
"trigger": {
|
||||
"schedule": {
|
||||
"interval": "1h"
|
||||
}
|
||||
},
|
||||
--------------------------------------------------
|
||||
// NOTCONSOLE
|
||||
--
|
||||
. Load data into the watch payload by creating an input that searches the meetup data for events that have _Open Source_ as a topic. You can use aggregations to group the data by city, consolidate references to the same events, and sort the events by date.
|
||||
+
|
||||
--
|
||||
[source,js]
|
||||
-------------------------------------------------
|
||||
"input": {
|
||||
"search": {
|
||||
"request": {
|
||||
"indices": [
|
||||
"logstash" <1>
|
||||
],
|
||||
"body": {
|
||||
"size": 0,
|
||||
"query": {
|
||||
"bool": {
|
||||
"filter": [
|
||||
{
|
||||
"range": {
|
||||
"@timestamp": {
|
||||
"gte": "now-3h"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"match": {
|
||||
"group.group_topics.topic_name": "Open Source" <2>
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"aggs": {
|
||||
"group_by_city": {
|
||||
"terms": {
|
||||
"field": "group.group_city.keyword", <3>
|
||||
"size": 5
|
||||
},
|
||||
"aggs": {
|
||||
"group_by_event": {
|
||||
"terms": {
|
||||
"field": "event.event_url.keyword", <4>
|
||||
"size": 5
|
||||
},
|
||||
"aggs": {
|
||||
"get_latest": {
|
||||
"terms": {
|
||||
"field": "@timestamp", <5>
|
||||
"size": 1,
|
||||
"order": {
|
||||
"_key": "desc"
|
||||
}
|
||||
},
|
||||
"aggs": {
|
||||
"group_by_event_name": {
|
||||
"terms": {
|
||||
"field": "event.event_name.keyword" <6>
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
-------------------------------------------------
|
||||
// NOTCONSOLE
|
||||
<1> `logstash` is the default <<aliases,index alias>> for the {ls} indices
|
||||
containing the meetup data. By default, the {ls}
|
||||
<<index-lifecycle-management,{ilm} ({ilm-init})>> policy rolls this alias to a
|
||||
new index when the index size reaches 50GB or becomes 30 days old. For more
|
||||
information, see
|
||||
{logstash-ref}/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-ilm[{ilm-init}
|
||||
defaults in Logstash].
|
||||
<2> Find all of the RSVPs with `Open Source` as a topic.
|
||||
<3> Group the RSVPs by city.
|
||||
<4> Consolidate multiple RSVPs for the same event.
|
||||
<5> Sort the events so the latest events are listed first.
|
||||
<6> Group the events by name.
|
||||
--
|
||||
|
||||
. To determine whether or not there are any Open Source events, add a compare condition that checks the watch payload to see if there were any search hits.
|
||||
+
|
||||
--
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
"compare" : { "ctx.payload.hits.total" : { "gt" : 0 }}
|
||||
--------------------------------------------------
|
||||
// NOTCONSOLE
|
||||
--
|
||||
|
||||
. To send an email when _Open Source_ events are found, add an email action:
|
||||
+
|
||||
--
|
||||
[source,js]
|
||||
---------------------------------------------------
|
||||
"actions": {
|
||||
"email_me": {
|
||||
"throttle_period": "10m",
|
||||
"email": {
|
||||
"from": "<from:email address>",
|
||||
"to": "<to:email address>",
|
||||
"subject": "Open Source Events",
|
||||
"body": {
|
||||
"html": "Found events matching Open Source: <ul>{{#ctx.payload.aggregations.group_by_city.buckets}}< li>{{key}} ({{doc_count}})<ul>{{#group_by_event.buckets}}
|
||||
<li><a href=\"{{key}}\">{{get_latest.buckets.0.group_by_event_name.buckets.0.key}}</a>
|
||||
({{doc_count}})</li>{{/group_by_event.buckets}}</ul></li>
|
||||
{{/ctx.payload.aggregations.group_by_city.buckets}}</ul>"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
---------------------------------------------------
|
||||
// NOTCONSOLE
|
||||
--
|
||||
|
||||
NOTE: To enable Watcher to send emails, you must configure an email account in `elasticsearch.yml`. For more information, see <<configuring-email>>.
|
||||
|
||||
|
||||
The complete watch looks like this:
|
||||
|
||||
[source,console]
|
||||
--------------------------------------------------
|
||||
PUT _watcher/watch/meetup
|
||||
{
|
||||
"trigger": {
|
||||
"schedule": {
|
||||
"interval": "1h"
|
||||
}
|
||||
},
|
||||
"input": {
|
||||
"search": {
|
||||
"request": {
|
||||
"indices": [
|
||||
"logstash"
|
||||
],
|
||||
"body": {
|
||||
"size": 0,
|
||||
"query": {
|
||||
"bool": {
|
||||
"filter": [
|
||||
{
|
||||
"range": {
|
||||
"@timestamp": {
|
||||
"gte": "now-3h"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"match": {
|
||||
"group.group_topics.topic_name": "Open Source"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"aggs": {
|
||||
"group_by_city": {
|
||||
"terms": {
|
||||
"field": "group.group_city.keyword",
|
||||
"size": 5
|
||||
},
|
||||
"aggs": {
|
||||
"group_by_event": {
|
||||
"terms": {
|
||||
"field": "event.event_url.keyword",
|
||||
"size": 5
|
||||
},
|
||||
"aggs": {
|
||||
"get_latest": {
|
||||
"terms": {
|
||||
"field": "@timestamp",
|
||||
"size": 1,
|
||||
"order": {
|
||||
"_key": "desc"
|
||||
}
|
||||
},
|
||||
"aggs": {
|
||||
"group_by_event_name": {
|
||||
"terms": {
|
||||
"field": "event.event_name.keyword"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"condition": {
|
||||
"compare": {
|
||||
"ctx.payload.hits.total": {
|
||||
"gt": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"actions": { <1>
|
||||
"email_me": {
|
||||
"throttle_period": "10m",
|
||||
"email": {
|
||||
"from": "username@example.org", <2>
|
||||
"to": "recipient@example.org", <3>
|
||||
"subject": "Open Source events",
|
||||
"body": {
|
||||
"html": "Found events matching Open Source: <ul>{{#ctx.payload.aggregations.group_by_city.buckets}}<li>{{key}} ({{doc_count}})<ul>{{#group_by_event.buckets}}<li><a href=\"{{key}}\">{{get_latest.buckets.0.group_by_event_name.buckets.0.key}}</a> ({{doc_count}})</li>{{/group_by_event.buckets}}</ul></li>{{/ctx.payload.aggregations.group_by_city.buckets}}</ul>"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
--------------------------------------------------
|
||||
|
||||
<1> The email body can include Mustache templates to reference data in the watch payload. By default,it will be <<email-html-sanitization,sanitized>> to block dangerous content.
|
||||
<2> Replace the `from` address with the email address you configured in `elasticsearch.yml`.
|
||||
<3> Replace the `to` address with your email address to receive notifications.
|
||||
|
||||
|
||||
Now that you've created your watch, you can use the
|
||||
{ref}/watcher-api-execute-watch.html[`_execute` API] to run it without waiting for the schedule to trigger execution:
|
||||
|
||||
[source,console]
|
||||
--------------------------------------------------
|
||||
POST _watcher/watch/meetup/_execute
|
||||
--------------------------------------------------
|
||||
// TEST[continued]
|
||||
Loading…
Add table
Add a link
Reference in a new issue