mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-04-25 07:37:19 -04:00
c1b39322af
Currently the documentation on network threading suggests that we still use a model where we have individual workers dedicated to server sockets. That is no longer true and server sockets are assigned to normal workers. This commit updates the documentation.
111 lines
6.3 KiB
Text
111 lines
6.3 KiB
Text
[[modules-network-threading-model]]
|
|
==== Networking threading model
|
|
|
|
This section describes the threading model used by the networking subsystem in
|
|
{es}. This information isn't required to use {es}, but it may be useful to
|
|
advanced users who are diagnosing network problems in a cluster.
|
|
|
|
{es} nodes communicate over a collection of TCP channels that together form a
|
|
transport connection. {es} clients communicate with the cluster over HTTP,
|
|
which also uses one or more TCP channels. Each of these TCP channels is owned
|
|
by exactly one of the `transport_worker` threads in the node. This owning
|
|
thread is chosen when the channel is opened and remains the same for the
|
|
lifetime of the channel.
|
|
|
|
Each `transport_worker` thread has sole responsibility for sending and
|
|
receiving data over the channels it owns. Additionally, each http and transport
|
|
server socket is assigned to one of the `transport_worker` threads. That worker
|
|
has the responsibility of accepting new incoming connections to the server
|
|
socket it owns.
|
|
|
|
If a thread in {es} wants to send data over a particular channel, it passes the
|
|
data to the owning `transport_worker` thread for the actual transmission.
|
|
|
|
Normally the `transport_worker` threads will not completely handle the messages
|
|
they receive. Instead, they will do a small amount of preliminary processing
|
|
and then dispatch (hand off) the message to a different
|
|
<<modules-threadpool,threadpool>> for the rest of their handling. For instance,
|
|
bulk messages are dispatched to the `write` threadpool, searches are dispatched
|
|
to one of the `search` threadpools, and requests for statistics and other
|
|
management tasks are mostly dispatched to the `management` threadpool. However
|
|
in some cases the processing of a message is expected to be so quick that {es}
|
|
will do all of the processing on the `transport_worker` thread rather than
|
|
incur the overhead of dispatching it elsewhere.
|
|
|
|
By default, there is one `transport_worker` thread per CPU. In contrast, there
|
|
may sometimes be tens-of-thousands of TCP channels. If data arrives on a TCP
|
|
channel and its owning `transport_worker` thread is busy, the data isn't
|
|
processed until the thread finishes whatever it is doing. Similarly, outgoing
|
|
data are not sent over a channel until the owning `transport_worker` thread is
|
|
free. This means that we require every `transport_worker` thread to be idle
|
|
frequently. An idle `transport_worker` looks something like this in a stack
|
|
dump:
|
|
|
|
[source,text]
|
|
----
|
|
"elasticsearch[instance-0000000004][transport_worker][T#1]" #32 daemon prio=5 os_prio=0 cpu=9645.94ms elapsed=501.63s tid=0x00007fb83b6307f0 nid=0x1c4 runnable [0x00007fb7b8ffe000]
|
|
java.lang.Thread.State: RUNNABLE
|
|
at sun.nio.ch.EPoll.wait(java.base@17.0.2/Native Method)
|
|
at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@17.0.2/EPollSelectorImpl.java:118)
|
|
at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@17.0.2/SelectorImpl.java:129)
|
|
- locked <0x00000000c443c518> (a sun.nio.ch.Util$2)
|
|
- locked <0x00000000c38f7700> (a sun.nio.ch.EPollSelectorImpl)
|
|
at sun.nio.ch.SelectorImpl.select(java.base@17.0.2/SelectorImpl.java:146)
|
|
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:813)
|
|
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
|
|
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
|
|
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
|
|
at java.lang.Thread.run(java.base@17.0.2/Thread.java:833)
|
|
----
|
|
|
|
In the <<cluster-nodes-hot-threads>> API an idle `transport_worker` thread is
|
|
reported like this:
|
|
|
|
[source,text]
|
|
----
|
|
100.0% [cpu=0.0%, other=100.0%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[instance-0000000004][transport_worker][T#1]'
|
|
10/10 snapshots sharing following 9 elements
|
|
java.base@17.0.2/sun.nio.ch.EPoll.wait(Native Method)
|
|
java.base@17.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:118)
|
|
java.base@17.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:129)
|
|
java.base@17.0.2/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:146)
|
|
io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:813)
|
|
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
|
|
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
|
|
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
|
|
java.base@17.0.2/java.lang.Thread.run(Thread.java:833)
|
|
----
|
|
|
|
Note that `transport_worker` threads should always be in state `RUNNABLE`, even
|
|
when waiting for input, because they block in the native `EPoll#wait` method.
|
|
This means the hot threads API will report these threads at 100% overall
|
|
utilisation. This is normal, and the breakdown of time into `cpu=` and `other=`
|
|
fractions shows how much time the thread spent running and waiting for input
|
|
respectively.
|
|
|
|
If a `transport_worker` thread is not frequently idle, it may build up a
|
|
backlog of work. This can cause delays in processing messages on the channels
|
|
that it owns. It's hard to predict exactly which work will be delayed:
|
|
|
|
* There are many more channels than threads. If work related to one channel is
|
|
causing delays to its worker thread, all other channels owned by that thread
|
|
will also suffer delays.
|
|
|
|
* The mapping from TCP channels to worker threads is fixed but arbitrary. Each
|
|
channel is assigned an owning thread in a round-robin fashion when the channel
|
|
is opened. Each worker thread is responsible for many different kinds of
|
|
channel.
|
|
|
|
* There are many channels open between each pair of nodes. For each request,
|
|
{es} will choose from the appropriate channels in a round-robin fashion. Some
|
|
requests may end up on a channel owned by a delayed worker while other
|
|
identical requests will be sent on a channel that's working smoothly.
|
|
|
|
If the backlog builds up too far, some messages may be delayed by many seconds.
|
|
The node might even <<cluster-fault-detection,fail its health checks>> and be
|
|
removed from the cluster. Sometimes, you can find evidence of busy
|
|
`transport_worker` threads using the <<cluster-nodes-hot-threads>> API.
|
|
However, this API itself sends network messages so may not work correctly if
|
|
the `transport_worker` threads are too busy. It is more reliable to use
|
|
`jstack` to obtain stack dumps or use Java Flight Recorder to obtain a
|
|
profiling trace. These tools are independent of any work the JVM is performing.
|