mirror of
https://github.com/elastic/kibana.git
synced 2025-04-23 17:28:26 -04:00
[DOCS] Updates docs for Grok Debugger (#36289)
This commit is contained in:
gchaps
Sarah Hersh
parent
279c7cea3e
commit
883ff0c26b
8 changed files with 107 additions and 137 deletions
|
|
@ -1,124 +0,0 @@
|
|||
[role="xpack"]
|
||||
[[grokdebugger-getting-started]]
|
||||
|
||||
ifndef::gs-mini[]
|
||||
=== Getting Started with the Grok Debugger
|
||||
endif::gs-mini[]
|
||||
|
||||
ifdef::gs-mini[]
|
||||
== Getting Started with the Grok Debugger
|
||||
endif::gs-mini[]
|
||||
|
||||
++++
|
||||
<titleabbrev>Getting Started</titleabbrev>
|
||||
++++
|
||||
|
||||
TIP: See the documentation about the ingest node
|
||||
{ref}/grok-processor.html[grok processor] and the Logstash {logstash-ref}/plugins-filters-grok.html[grok filter] more info about grok.
|
||||
|
||||
NOTE: If you're using {security}, you must have the `manage_pipeline`
|
||||
permission in order to use the Grok Debugger.
|
||||
|
||||
The Grok Debugger is automatically enabled in {kib}. It is located under the *DevTools* tab in {kib}.
|
||||
|
||||
To start debugging grok patterns:
|
||||
|
||||
. Open Kibana in your web browser and log in. If you are running Kibana
|
||||
locally, go to `http://localhost:5601/`.
|
||||
|
||||
. Click **DevTools** in the side navigation and then click **Grok Debugger**
|
||||
on the top navigation bar.
|
||||
+
|
||||
image::dev-tools/grokdebugger/images/grok-debugger.png["Grok Debugger UI"]
|
||||
|
||||
. Under Sample Data, enter a sample message that is representative of the data you
|
||||
want to parse. For example:
|
||||
+
|
||||
[source,ruby]
|
||||
-------------------------------------------------------------------------------
|
||||
55.3.244.1 GET /index.html 15824 0.043
|
||||
-------------------------------------------------------------------------------
|
||||
|
||||
. Under Grok Pattern, enter the grok pattern that you want to apply to the data.
|
||||
+
|
||||
For example, to parse the log line in the example, you use:
|
||||
+
|
||||
[source,ruby]
|
||||
-------------------------------------------------------------------------------
|
||||
%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}
|
||||
-------------------------------------------------------------------------------
|
||||
|
||||
. Click the **Simulate** button.
|
||||
+
|
||||
Under Structured Data, you'll see the simulated event that results from applying the grok
|
||||
pattern:
|
||||
+
|
||||
image::dev-tools/grokdebugger/images/grok-debugger-output.png["Viewing Grok Debugger Output"]
|
||||
+
|
||||
Any errors in the pattern will appear at the top of the page. For example,
|
||||
here you see a parse exception because the pattern name is misspelled as `WORDD`
|
||||
and therefore can't be found in the pattern dictionary:
|
||||
+
|
||||
image::dev-tools/grokdebugger/images/grok-debugger-error.png["Viewing Grok Debugger Errors"]
|
||||
+
|
||||
You can click the **More** link to see more detail about the message.
|
||||
+
|
||||
Click **OK** to dismiss the message and continue iterating over the grok pattern
|
||||
until there are no errors and the output matches the event that you expect.
|
||||
|
||||
//TODO: Update LS and ingest node docs with pointers to the new grok debugger. Replace references to the Heroku app.
|
||||
|
||||
[float]
|
||||
[[grokdebugger-custom-patterns]]
|
||||
==== Testing Custom Patterns
|
||||
|
||||
If the default grok pattern dictionary doesn't contain the patterns you need,
|
||||
you may need to define custom patterns. You can use the Grok Debugger to test
|
||||
and debug customer patterns.
|
||||
|
||||
The custom patterns that you enter in the Grok Debugger are not saved. They're
|
||||
only available for the current debugging session and have no side effects.
|
||||
|
||||
To test a custom pattern:
|
||||
|
||||
. Repeat the steps that you followed previously to enter the sample message and
|
||||
grok pattern. For this example, let's use the following sample message:
|
||||
+
|
||||
[source,ruby]
|
||||
-------------------------------------------------------------------------------
|
||||
Jan 1 06:25:43 mailserver14 postfix/cleanup[21403]: BEF25A72965: message-id=<20130101142543.5828399CCAF@mailserver14.example.com>
|
||||
-------------------------------------------------------------------------------
|
||||
+
|
||||
And this grok pattern:
|
||||
+
|
||||
[source,ruby]
|
||||
-------------------------------------------------------------------------------
|
||||
%{SYSLOGBASE} %{POSTFIX_QUEUEID:queue_id}: %{MSG:syslog_message}
|
||||
-------------------------------------------------------------------------------
|
||||
+
|
||||
Notice that the grok pattern references custom patterns called `POSTFIX_QUEUEID`
|
||||
and `MSG`.
|
||||
|
||||
. Expand **Custom Patterns** and enter pattern definitions for any custom
|
||||
patterns that you want to use in the grok expression. Each pattern definition
|
||||
must be specified on its own line.
|
||||
+
|
||||
For the grok pattern in the example, you need to specify pattern definitions
|
||||
for `POSTFIX_QUEUEID` and `MSG`:
|
||||
+
|
||||
[source,ruby]
|
||||
-------------------------------------------------------------------------------
|
||||
POSTFIX_QUEUEID [0-9A-F]{10,11}
|
||||
MSG message-id=<%{GREEDYDATA}>
|
||||
-------------------------------------------------------------------------------
|
||||
|
||||
. Click the **Simulate** button.
|
||||
+
|
||||
Under Output, you'll see the simulated output event that results from applying
|
||||
the grok pattern that contains the custom pattern:
|
||||
+
|
||||
image::dev-tools/grokdebugger/images/grok-debugger-custom-pattern.png["Debugging a custom pattern"]
|
||||
+
|
||||
If an error occurs, you can view the error message and continue iterating over
|
||||
the custom pattern until there are no errors and the output matches the event
|
||||
that you expect.
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 348 KiB After Width: | Height: | Size: 195 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 211 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 280 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 138 KiB After Width: | Height: | Size: 109 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 68 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 208 KiB |
|
|
@ -1,25 +1,119 @@
|
|||
[role="xpack"]
|
||||
[[xpack-grokdebugger]]
|
||||
== Debugging Grok Expressions
|
||||
== Debugging grok expressions
|
||||
|
||||
Grok is a pattern matching syntax that you can use to parse arbitrary text and
|
||||
structure it. Grok is perfect for parsing syslog logs, apache and other
|
||||
webserver logs, mysql logs, and in general, any log format that is generally
|
||||
written for humans and not computer consumption.
|
||||
You can build and debug grok patterns in the {kib} *Grok Debugger*
|
||||
before you use them in your data processing pipelines. Grok is a pattern
|
||||
matching syntax that you can use to parse arbitrary text and
|
||||
structure it. Grok is good for parsing syslog, apache, and other
|
||||
webserver logs, mysql logs, and in general, any log format that is
|
||||
written for human consumption.
|
||||
|
||||
Grok patterns are supported in the ingest node
|
||||
{ref}/grok-processor.html[grok processor] and the Logstash
|
||||
{logstash-ref}/plugins-filters-grok.html[grok filter]. The Elastic Stack ships
|
||||
with over 120 reusable grok patterns. See
|
||||
https://github.com/elastic/elasticsearch/tree/master/modules/ingest-common/src/main/resources/patterns[Ingest node grok patterns] and https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns[Logstash grok patterns]
|
||||
for the full list of patterns.
|
||||
{logstash-ref}/plugins-filters-grok.html[grok filter]. See
|
||||
{logstash-ref}/plugins-filters-grok.html#_grok_basics[grok basics]
|
||||
for more information on the syntax for a grok pattern.
|
||||
|
||||
You can build and debug grok patterns in the Grok Debugger tool in {kib}
|
||||
before you use them in your data processing pipelines. Because
|
||||
The Elastic Stack ships
|
||||
with more than 120 reusable grok patterns. See
|
||||
https://github.com/elastic/elasticsearch/tree/master/libs/grok/src/main/resources/patterns[Ingest node grok patterns] and https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns[Logstash grok patterns]
|
||||
for the complete list of patterns.
|
||||
|
||||
Because
|
||||
ingest node and Logstash share the same grok implementation and pattern
|
||||
libraries, any grok pattern that you create in the Grok Debugger will work
|
||||
libraries, any grok pattern that you create in the *Grok Debugger* will work
|
||||
in ingest node and Logstash.
|
||||
|
||||
[float]
|
||||
[[grokdebugger-getting-started]]
|
||||
=== Getting started with the Grok Debugger
|
||||
|
||||
This example walks you through using the *Grok Debugger*. This tool
|
||||
is automatically enabled in {kib}.
|
||||
|
||||
NOTE: If you're using {security}, you must have the `manage_pipeline`
|
||||
permission to use the Grok Debugger.
|
||||
|
||||
. In the side navigation, click *DevTools*, then open the *Grok Debugger*.
|
||||
. In *Sample Data*, enter a message that is representative of the data that you
|
||||
want to parse. For example:
|
||||
+
|
||||
[source,ruby]
|
||||
-------------------------------------------------------------------------------
|
||||
55.3.244.1 GET /index.html 15824 0.043
|
||||
-------------------------------------------------------------------------------
|
||||
|
||||
. In *Grok Pattern*, enter the grok pattern that you want to apply to the data.
|
||||
+
|
||||
To parse the log line in this example, use:
|
||||
+
|
||||
[source,ruby]
|
||||
-------------------------------------------------------------------------------
|
||||
%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}
|
||||
-------------------------------------------------------------------------------
|
||||
|
||||
. Click **Simulate**.
|
||||
+
|
||||
You'll see the simulated event that results from applying the grok
|
||||
pattern.
|
||||
+
|
||||
[role="screenshot"]
|
||||
image::dev-tools/grokdebugger/images/grok-debugger-overview.png["Grok Debugger"]
|
||||
|
||||
include::getting-started.asciidoc[]
|
||||
|
||||
//TODO: Update LS and ingest node docs with pointers to the new grok debugger. Replace references to the Heroku app.
|
||||
|
||||
[float]
|
||||
[[grokdebugger-custom-patterns]]
|
||||
=== Testing custom patterns
|
||||
|
||||
If the default grok pattern dictionary doesn't contain the patterns you need,
|
||||
you can define, test, and debug custom patterns using the Grok Debugger.
|
||||
|
||||
Custom patterns that you enter in the Grok Debugger are not saved. Custom patterns
|
||||
are only available for the current debugging session and have no side effects.
|
||||
|
||||
Follow this example to define a custom pattern.
|
||||
|
||||
. In *Sample Data*, enter the following sample message:
|
||||
+
|
||||
[source,ruby]
|
||||
-------------------------------------------------------------------------------
|
||||
Jan 1 06:25:43 mailserver14 postfix/cleanup[21403]: BEF25A72965: message-id=<20130101142543.5828399CCAF@mailserver14.example.com>
|
||||
-------------------------------------------------------------------------------
|
||||
|
||||
. Enter this grok pattern:
|
||||
+
|
||||
[source,ruby]
|
||||
-------------------------------------------------------------------------------
|
||||
%{SYSLOGBASE} %{POSTFIX_QUEUEID:queue_id}: %{MSG:syslog_message}
|
||||
-------------------------------------------------------------------------------
|
||||
+
|
||||
Notice that the grok pattern references custom patterns called `POSTFIX_QUEUEID`
|
||||
and `MSG`.
|
||||
|
||||
. Expand **Custom Patterns** and enter pattern definitions for the custom
|
||||
patterns that you want to use in the grok expression. You must specify each pattern definition
|
||||
on its own line.
|
||||
+
|
||||
For this example, you must specify pattern definitions
|
||||
for `POSTFIX_QUEUEID` and `MSG`:
|
||||
+
|
||||
[source,ruby]
|
||||
-------------------------------------------------------------------------------
|
||||
POSTFIX_QUEUEID [0-9A-F]{10,11}
|
||||
MSG message-id=<%{GREEDYDATA}>
|
||||
-------------------------------------------------------------------------------
|
||||
|
||||
. Click **Simulate**.
|
||||
+
|
||||
You'll see the simulated output event that results from applying
|
||||
the grok pattern that contains the custom pattern:
|
||||
+
|
||||
[role="screenshot"]
|
||||
image::dev-tools/grokdebugger/images/grok-debugger-custom-pattern.png["Debugging a custom pattern"]
|
||||
+
|
||||
If an error occurs, you can continue iterating over
|
||||
the custom pattern until the output matches the event
|
||||
that you expect.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue