[[monitoring]] == Monitoring APIs experimental[] Logstash provides the following monitoring APIs to retrieve runtime metrics about Logstash: * <> * <> * <> * <> You can use the root resource to retrieve general information about the Logstash instance, including the host and version. [source,js] -------------------------------------------------- GET / -------------------------------------------------- Example response: ["source","js",subs="attributes"] -------------------------------------------------- { "host": "skywalker", "version": "{logstash_version}", "http_address": "127.0.0.1:9600" } -------------------------------------------------- NOTE: By default, the monitoring API attempts to bind to `tcp:9600`. If this port is already in use by another Logstash instance, you need to launch Logstash with the `--http.port` flag specified to bind to a different port. See <> for more information. [float] [[monitoring-common-options]] === Common Options The following options can be applied to all of the Logstash monitoring APIs. [float] ==== Pretty Results When appending `?pretty=true` to any request made, the JSON returned will be pretty formatted (use it for debugging only!). [float] ==== Human-Readable Output NOTE: For Logstash {logstash_version}, the `human` option is supported for the <> only. When you specify `human=true`, the results are returned in plain text instead of JSON format. The default is false. Statistics are returned in a format suitable for humans (eg `"exists_time": "1h"` or `"size": "1kb"`) and for computers (eg `"exists_time_in_millis": 3600000` or `"size_in_bytes": 1024`). The human-readable values can be turned off by adding `?human=false` to the query string. This makes sense when the stats results are being consumed by a monitoring tool, rather than intended for human consumption. The default for the `human` flag is `false`. [[node-info-api]] === Node Info API experimental[] The node info API retrieves information about the node. [source,js] -------------------------------------------------- GET /_node/ -------------------------------------------------- Where `` is optional and specifies the types of node info you want to return. You can limit the info that's returned by combining any of the following types in a comma-separated list: [horizontal] <>:: Gets pipeline-specific information and settings. <>:: Gets node-level info about the OS. <>:: Gets node-level JVM info, including info about threads. See <> for a list of options that can be applied to all Logstash monitoring APIs. [[node-pipeline-info]] ==== Pipeline Info The following request returns a JSON document that shows pipeline info, such as the number of workers, batch size, and batch delay: [source,js] -------------------------------------------------- GET /_node/pipeline -------------------------------------------------- If you want to view additional information about the pipeline, such as stats for each configured input, filter, or output stage, see the <> section under the <>. Example response: ["source","js",subs="attributes"] -------------------------------------------------- { "pipeline": { "workers": 8, "batch_size": 125, "batch_delay": 5, "config_reload_automatic": true, "config_reload_interval": 3 } -------------------------------------------------- [[node-os-info]] ==== OS Info The following request returns a JSON document that shows the OS name, architecture, version, and available processors: [source,js] -------------------------------------------------- GET /_node/os -------------------------------------------------- Example response: [source,js] -------------------------------------------------- { "os": { "name": "Mac OS X", "arch": "x86_64", "version": "10.12.1", "available_processors": 8 } -------------------------------------------------- [[node-jvm-info]] ==== JVM Info The following request returns a JSON document that shows node-level JVM stats, such as the JVM process id, version, VM info, memory usage, and info about garbage collectors: [source,js] -------------------------------------------------- GET /_node/jvm -------------------------------------------------- Example response: [source,js] -------------------------------------------------- { "jvm": { "pid": 59616, "version": "1.8.0_65", "vm_name": "Java HotSpot(TM) 64-Bit Server VM", "vm_version": "1.8.0_65", "vm_vendor": "Oracle Corporation", "start_time_in_millis": 1484251185878, "mem": { "heap_init_in_bytes": 268435456, "heap_max_in_bytes": 1037959168, "non_heap_init_in_bytes": 2555904, "non_heap_max_in_bytes": 0 }, "gc_collectors": [ "ParNew", "ConcurrentMarkSweep" ] } } -------------------------------------------------- [[plugins-api]] === Plugins Info API experimental[] The plugins info API gets information about all Logstash plugins that are currently installed. This API basically returns the output of running the `bin/logstash-plugin list --verbose` command. [source,js] -------------------------------------------------- GET /_node/plugins -------------------------------------------------- See <> for a list of options that can be applied to all Logstash monitoring APIs. The output is a JSON document. Example response: ["source","js",subs="attributes"] -------------------------------------------------- { "total": 92, "plugins": [ { "name": "logstash-codec-cef", "version": "4.1.2" }, { "name": "logstash-codec-collectd", "version": "3.0.3" }, { "name": "logstash-codec-dots", "version": "3.0.2" }, { "name": "logstash-codec-edn", "version": "3.0.2" }, . . . ] -------------------------------------------------- [[node-stats-api]] === Node Stats API experimental[] The node stats API retrieves runtime stats about Logstash. [source,js] -------------------------------------------------- GET /_node/stats/ -------------------------------------------------- Where `` is optional and specifies the types of stats you want to return. By default, all stats are returned. You can limit the info that's returned by combining any of the following types in a comma-separated list: [horizontal] <>:: Gets JVM stats, including stats about threads, memory usage, garbage collectors, and uptime. <>:: Gets process stats, including stats about file descriptors, memory consumption, and CPU usage. <>:: Gets runtime stats about the Logstash pipeline. <>:: Gets runtime stats about config reload successes and failures. <>:: Gets runtime stats about cgroups when Logstash is running in a container. See <> for a list of options that can be applied to all Logstash monitoring APIs. [[jvm-stats]] ==== JVM Stats The following request returns a JSON document containing JVM stats: [source,js] -------------------------------------------------- GET /_node/stats/jvm -------------------------------------------------- Example response: [source,js] -------------------------------------------------- { "jvm": { "threads": { "count": 35, "peak_count": 36 }, "mem": { "heap_used_in_bytes": 318691184, "heap_used_percent": 15, "heap_committed_in_bytes": 519045120, "heap_max_in_bytes": 2075918336, "non_heap_used_in_bytes": 189382304, "non_heap_committed_in_bytes": 200728576, "pools": { "survivor": { "peak_used_in_bytes": 8912896, "used_in_bytes": 9538656, "peak_max_in_bytes": 35782656, "max_in_bytes": 71565312, "committed_in_bytes": 17825792 }, "old": { "peak_used_in_bytes": 106946320, "used_in_bytes": 181913072, "peak_max_in_bytes": 715849728, "max_in_bytes": 1431699456, "committed_in_bytes": 357957632 }, "young": { "peak_used_in_bytes": 71630848, "used_in_bytes": 127239456, "peak_max_in_bytes": 286326784, "max_in_bytes": 572653568, "committed_in_bytes": 143261696 } } }, "gc": { "collectors": { "old": { "collection_time_in_millis": 58, "collection_count": 2 }, "young": { "collection_time_in_millis": 338, "collection_count": 26 } } }, "uptime_in_millis": 382701 } -------------------------------------------------- [[process-stats]] ==== Process Stats The following request returns a JSON document containing process stats: [source,js] -------------------------------------------------- GET /_node/stats/process -------------------------------------------------- Example response: [source,js] -------------------------------------------------- { "process": { "open_file_descriptors": 164, "peak_open_file_descriptors": 166, "max_file_descriptors": 10240, "mem": { "total_virtual_in_bytes": 5399474176 }, "cpu": { "total_in_millis": 72810537000, "percent": 0, "load_average": { "1m": 2.41943359375 } } } } -------------------------------------------------- [[pipeline-stats]] ==== Pipeline Stats The following request returns a JSON document containing pipeline stats, including: * the number of events that were input, filtered, or output by the pipeline * stats for each configured filter or output stage * info about config reload successes and failures (when <> is enabled) * info about the persistent queue (when <> are enabled) NOTE: Detailed pipeline stats for input plugins are not currently available, but will be available in a future release. For now, the node stats API returns an empty set array for inputs (`"inputs": []`). [source,js] -------------------------------------------------- GET /_node/stats/pipeline -------------------------------------------------- Example response: [source,js] -------------------------------------------------- { "pipeline": { "events": { "duration_in_millis": 6304989, "in": 200, "filtered": 200, "out": 200 }, "plugins": { "inputs": [], "filters": [ { "id": "4e3d4bed6ba821ebb47f4752bb757b04a754d736-2", "events": { "duration_in_millis": 113, "in": 200, "out": 200 }, "matches": 200, "patterns_per_field": { "message": 1 }, "name": "grok" }, { "id": "4e3d4bed6ba821ebb47f4752bb757b04a754d736-3", "events": { "duration_in_millis": 526, "in": 200, "out": 200 }, "name": "geoip" } ], "outputs": [ { "id": "4e3d4bed6ba821ebb47f4752bb757b04a754d736-4", "events": { "duration_in_millis": 2312, "in": 200, "out": 200 }, "name": "stdout" } ] }, "reloads": { "last_error": null, "successes": 0, "last_success_timestamp": null, "last_failure_timestamp": null, "failures": 0 }, "queue": { "events": 26, "type": "persisted", "capacity": { "page_capacity_in_bytes": 262144000, "max_queue_size_in_bytes": 4294967296, "max_unread_events": 0 }, "data": { "path": "/path/to/data/queue", "free_space_in_bytes": 123027787776, "storage_type": "hfs" } } } } -------------------------------------------------- [[reload-stats]] ==== Reload Stats The following request returns a JSON document that shows info about config reload successes and failures. [source,js] -------------------------------------------------- GET /_node/stats/reloads -------------------------------------------------- Example response: [source,js] -------------------------------------------------- { "reloads": { "successes": 0, "failures": 0 } } -------------------------------------------------- [[os-stats]] ==== OS Stats When Logstash is running in a container, the following request returns a JSON document that contains cgroup information to give you a more accurate view of CPU load, including whether the container is being throttled. [source,js] -------------------------------------------------- GET /_node/stats/os -------------------------------------------------- Example response: [source,js] -------------------------------------------------- { "os" : { "cgroup" : { "cpuacct" : { "control_group" : "/elastic1", "usage_nanos" : 378477588075 }, "cpu" : { "control_group" : "/elastic1", "cfs_period_micros" : 1000000, "cfs_quota_micros" : 800000, "stat" : { "number_of_elapsed_periods" : 4157, "number_of_times_throttled" : 460, "time_throttled_nanos" : 581617440755 } } } } -------------------------------------------------- [[hot-threads-api]] === Hot Threads API experimental[] The hot threads API gets the current hot threads for Logstash. A hot thread is a Java thread that has high CPU usage and executes for a longer than normal period of time. [source,js] -------------------------------------------------- GET /_node/hot_threads -------------------------------------------------- The output is a JSON document that contains a breakdown of the top hot threads for Logstash. Example response: [source,js] -------------------------------------------------- { "time": "2017-01-12T12:09:45-08:00", "busiest_threads": 3, "threads": [ { "name": "LogStash::Runner", "percent_of_cpu_time": 1.07, "state": "timed_waiting", "traces": [ "java.lang.Object.wait(Native Method)", "java.lang.Thread.join(Thread.java:1253)", "org.jruby.internal.runtime.NativeThread.join(NativeThread.java:75)", "org.jruby.RubyThread.join(RubyThread.java:697)", "org.jruby.RubyThread$INVOKER$i$0$1$join.call(RubyThread$INVOKER$i$0$1$join.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:663)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:198)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)", "org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60)" ] }, { "name": "[main]>worker7", "percent_of_cpu_time": 0.71, "state": "waiting", "traces": [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)", "java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)", "java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(AbstractQueuedSynchronizer.java:897)", "java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1222)", "java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)", "org.jruby.RubyThread.lockInterruptibly(RubyThread.java:1470)", "org.jruby.ext.thread.Mutex.lock(Mutex.java:91)", "org.jruby.ext.thread.Mutex.synchronize(Mutex.java:147)", "org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen)" ] }, { "name": "[main]>worker3", "percent_of_cpu_time": 0.71, "state": "waiting", "traces": [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)", "java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)", "java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(AbstractQueuedSynchronizer.java:897)", "java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1222)", "java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)", "org.jruby.RubyThread.lockInterruptibly(RubyThread.java:1470)", "org.jruby.ext.thread.Mutex.lock(Mutex.java:91)", "org.jruby.ext.thread.Mutex.synchronize(Mutex.java:147)", "org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen)" ] } ] } } -------------------------------------------------- The parameters allowed are: [horizontal] `threads`:: The number of hot threads to return. The default is 3. `human`:: If true, returns plain text instead of JSON format. The default is false. `ignore_idle_threads`:: If true, does not return idle threads. The default is true. See <> for a list of options that can be applied to all Logstash monitoring APIs. You can use the `?human` parameter to return the document in a human-readable format. [source,js] -------------------------------------------------- GET /_node/hot_threads?human=true -------------------------------------------------- Example of a human-readable response: [source,js] -------------------------------------------------- ::: {} Hot threads at 2017-01-12T12:10:15-08:00, busiestThreads=3: ================================================================================ 1.02 % of cpu usage, state: timed_waiting, thread name: 'LogStash::Runner' java.lang.Object.wait(Native Method) java.lang.Thread.join(Thread.java:1253) org.jruby.internal.runtime.NativeThread.join(NativeThread.java:75) org.jruby.RubyThread.join(RubyThread.java:697) org.jruby.RubyThread$INVOKER$i$0$1$join.call(RubyThread$INVOKER$i$0$1$join.gen) org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:663) org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:198) org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306) org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136) org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60) -------------------------------------------------------------------------------- 0.71 % of cpu usage, state: waiting, thread name: '[main]>worker7' sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(AbstractQueuedSynchronizer.java:897) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1222) java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) org.jruby.RubyThread.lockInterruptibly(RubyThread.java:1470) org.jruby.ext.thread.Mutex.lock(Mutex.java:91) org.jruby.ext.thread.Mutex.synchronize(Mutex.java:147) org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen) -------------------------------------------------------------------------------- 0.71 % of cpu usage, state: timed_waiting, thread name: '[main]>worker3' sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:497) org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:466) org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:324) --------------------------------------------------