mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-04-25 07:37:19 -04:00
d8056c8213
This adds support for setting the refresh request parameter to `wait_for` in the `index`, `delete`, `update`, and `bulk` APIs. When `refresh=wait_for` is set those APIs will not return until their results have been made visible to search by a refresh. Also it adds a `forced_refresh` field to the response of `index`, `delete`, `update`, and to each item in a bulk response. This will be true for requests with `?refresh` or `?refresh=true` and will be true for some requests (see below) with `refresh=wait_for` but ought to otherwise always be false. `refresh=wait_for` is implemented as a list of `Tuple<Translog.Location, Consumer<Boolean>>`s in the new `RefreshListeners` class that is managed by `IndexShard`. The dynamic, index scoped `index.max_refresh_listeners` setting controls a maximum number of listeners allowed in any shard. If more than that many listeners accumulate in the engine then a refresh will be forced, the thread that adds the listener will be blocked until the refresh completes, and then the listener will be called with a `forcedRefresh` flag so it knows that it was the "straw that broke the camel's back". These listeners are only used by `refresh=wait_for` and that flag manifests itself as `forced_refresh` being `true` in the response. About half of this change comes from piping async-ness down to the appropriate layer in a way that is compatible with the ongoing with with sequence ids. Closes #1063 You can look up the winding story of all the commits here: https://github.com/elastic/elasticsearch/pull/17986 Here are the commit messages in case they are intersting to you: commit59a753b891Author: Nik Everett <nik9000@gmail.com> Date: Mon Jun 6 10:18:23 2016 -0400 Replace a method reference with implementing an interface Saves a single allocation and forces more commonality between the WriteResults. commit31f7861a85Author: Nik Everett <nik9000@gmail.com> Date: Mon Jun 6 10:07:55 2016 -0400 Revert "Replace static method that takes consumer with delegate class that takes an interface" This reverts commit777e23a659. commit777e23a659Author: Nik Everett <nik9000@gmail.com> Date: Mon Jun 6 09:29:35 2016 -0400 Replace static method that takes consumer with delegate class that takes an interface Same number of allocations, much less code duplication. commit9b49a480caAuthor: Nik Everett <nik9000@gmail.com> Date: Mon Jun 6 08:25:38 2016 -0400 Patch from boaz commitc2bc36524fAuthor: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 14:46:27 2016 -0400 Fix docs After updating to master we are actually testing them. commit03975ac056Author: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 14:20:11 2016 -0400 Cleanup after merge from master commit9c9a1deb00Author: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 14:09:14 2016 -0400 Breaking changes notes commit1c3e64ae06Merge:9e63ad6f67e580Author: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 14:00:05 2016 -0400 Merge branch 'master' into block_until_refresh2 commit9e63ad6de5Author: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 13:21:27 2016 -0400 Test for TransportWriteAction commit522ecb59d3Author: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 10:30:18 2016 -0400 Document deprecation commit0cd67b947fAuthor: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 10:26:23 2016 -0400 Deprecate setRefresh(boolean) Users should use `setRefresh(RefreshPolicy)` instead. commitaeb1be3f2cAuthor: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 10:12:27 2016 -0400 Remove checkstyle suppression It is fixed commit00d09a9caaAuthor: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 10:08:28 2016 -0400 Improve comment commit788164b898Author: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 10:01:01 2016 -0400 S/ReplicatedWriteResponse/WriteResponse/ Now it lines up with WriteRequest. commitb74cf3fe77Author: Nik Everett <nik9000@gmail.com> Date: Wed Jun 1 18:27:52 2016 -0400 Preserve `?refresh` behavior `?refresh` means the same things as `?refresh=true`. commit30f972bdaeAuthor: Nik Everett <nik9000@gmail.com> Date: Wed Jun 1 17:39:05 2016 -0400 Handle hanging documents If a document is added to the index during a refresh we weren't properly firing its refresh listener. This happened because the way we detect whether a refresh makes something visible or not is imperfect. It is ok because it always errs on the side of thinking that something isn't yet visible. So when a document arrives during a refresh the refresh listeners won't think it made it into a refresh when, often, it does. The way we work around this is by telling Elasticsearch that it ought to trigger a refresh if there are any pending refresh listeners even if there aren't pending documents to update. Lucene short circuits the refresh so it doesn't take that much effort, but the refresh listeners still get the signal that a refresh has come in and they still pick up the change and notify the listener. This means that the time that a listener can wait is actually slightly longer than the refresh interval. commitd523b5702bAuthor: Nik Everett <nik9000@gmail.com> Date: Wed Jun 1 14:34:01 2016 -0400 Explain Integer.MAX_VALUE commit4ffb7c0e95Author: Nik Everett <nik9000@gmail.com> Date: Wed Jun 1 14:27:39 2016 -0400 Fire all refresh listeners in a single thread Rather than queueing a runnable each. commit19606ec3bbAuthor: Nik Everett <nik9000@gmail.com> Date: Wed Jun 1 14:09:52 2016 -0400 Assert translog ordering commit6bb4e5c75eAuthor: Nik Everett <nik9000@gmail.com> Date: Wed Jun 1 13:17:44 2016 -0400 Support null RefreshListeners in InternalEngine Just skip using it. commit74be1480d6Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 18:02:03 2016 -0400 Move funny ShardInfo hack for bulk into bulk This should make it easier to understand because it is closer to where it matters.... commit2b771f8dabAuthor: Nik Everett <nik9000@gmail.com> Date: Tue May 31 17:39:46 2016 -0400 Pull listener out into an inner class with javadoc and stuff commit058481ad72Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 17:33:42 2016 -0400 Fix javadoc links commitd2123b1cabAuthor: Nik Everett <nik9000@gmail.com> Date: Tue May 31 17:28:09 2016 -0400 Make more stuff final commit8453fc4f78Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 17:26:48 2016 -0400 Javadoc commitfb16d2fc70Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 16:14:48 2016 -0400 Rewrite refresh docs commit5797d1b1c4Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 15:02:34 2016 -0400 Fix forced_refresh flag It wasn't being set. commit43ce50a1deAuthor: Nik Everett <nik9000@gmail.com> Date: Tue May 31 14:02:56 2016 -0400 Delay translog sync and flush until after refresh The sync might have occurred for us during the refresh so we have less work to do. Maybe. commitbb2739202eAuthor: Nik Everett <nik9000@gmail.com> Date: Tue May 31 13:08:08 2016 -0400 Remove duplication in WritePrimaryResult and WriteReplicaResult commit2f579f89b4Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 12:19:05 2016 -0400 Clean up registration of RefreshListeners commit87ab6e60caAuthor: Nik Everett <nik9000@gmail.com> Date: Tue May 31 11:28:30 2016 -0400 Shorten lock time in RefreshListeners Also use null to represent no listeners rather than an empty list. This saves allocating a new ArrayList every refresh cycle on every index. commit0d49d9c572Author: Nik Everett <nik9000@gmail.com> Date: Tue May 24 10:46:18 2016 -0400 Flip relationship between RefreshListeners and Engine Now RefreshListeners comes to Engine from EngineConfig. commitb2704b8a39Author: Nik Everett <nik9000@gmail.com> Date: Tue May 24 09:37:58 2016 -0400 Remove unused imports Maybe I added them? commit04343a2264Author: Nik Everett <nik9000@gmail.com> Date: Tue May 24 09:37:52 2016 -0400 Javadoc commitda1e765678Author: Nik Everett <nik9000@gmail.com> Date: Tue May 24 09:26:35 2016 -0400 Reply with non-null Also move the fsync and flush to before the refresh listener stuff. commit5d8eecd0d9Author: Nik Everett <nik9000@gmail.com> Date: Tue May 24 08:58:47 2016 -0400 Remove funky synchronization in AsyncReplicaAction commit1ec71eea0fAuthor: Nik Everett <nik9000@gmail.com> Date: Tue May 24 08:01:14 2016 -0400 s/LinkedTransferQueue/ArrayList/ commit7da36a4ceeAuthor: Nik Everett <nik9000@gmail.com> Date: Tue May 24 07:46:38 2016 -0400 More cleanup for RefreshListeners commit957e9b7700Author: Nik Everett <nik9000@gmail.com> Date: Tue May 24 07:34:13 2016 -0400 /Consumer<Runnable>/Executor/ commit4d8bf5d4a7Author: Nik Everett <nik9000@gmail.com> Date: Mon May 23 22:20:42 2016 -0400 explain commit15d948a348Author: Nik Everett <nik9000@gmail.com> Date: Mon May 23 22:17:59 2016 -0400 Better.... commitdc28951d02Author: Nik Everett <nik9000@gmail.com> Date: Mon May 23 21:09:20 2016 -0400 Javadocs and compromises commit8eebaa89c0Author: Nik Everett <nik9000@gmail.com> Date: Mon May 23 20:52:49 2016 -0400 Take boaz's changes to their logic conclusion and unbreak important stuff like bulk commit7056b96ea4Author: Nik Everett <nik9000@gmail.com> Date: Mon May 23 15:49:32 2016 -0400 Patch from boaz commit87be7eaed0Author: Nik Everett <nik9000@gmail.com> Date: Mon May 23 15:49:13 2016 -0400 Revert "Move async parts of replica operation outside of the lock" This reverts commit13807ad10b. commit13807ad10bAuthor: Nik Everett <nik9000@gmail.com> Date: Fri May 20 22:53:15 2016 -0400 Move async parts of replica operation outside of the lock commitb8cadcef56Author: Nik Everett <nik9000@gmail.com> Date: Fri May 20 16:17:20 2016 -0400 Docs commit91149e0580Author: Nik Everett <nik9000@gmail.com> Date: Fri May 20 15:17:40 2016 -0400 Finally! commit1ff50c2fafAuthor: Nik Everett <nik9000@gmail.com> Date: Fri May 20 15:01:53 2016 -0400 Remove Translog#lastWriteLocation I wasn't being careful enough with locks so it wasn't right anyway. Instead this builds a synthetic Tranlog.Location when you call getWriteLocation with much more relaxed equality guarantees. Rather than being equal to the last Translog.Location returned it is simply guaranteed to be greater than the last translog returned and less than the next. commit55596ea68bAuthor: Nik Everett <nik9000@gmail.com> Date: Fri May 20 14:40:06 2016 -0400 Remove listener from shardOperationOnPrimary Create instead asyncShardOperationOnPrimary which is called after all of the replica operations are started to handle any async operations. commit3322e26211Author: Nik Everett <nik9000@gmail.com> Date: Tue May 17 17:20:02 2016 -0400 Increase default maximum number of listeners to 1000 commit88171a8322Author: Nik Everett <nik9000@gmail.com> Date: Tue May 17 16:40:57 2016 -0400 Rename test commit179c27c4f8Author: Nik Everett <nik9000@gmail.com> Date: Tue May 17 16:35:27 2016 -0400 Move refresh listeners into their own class They still live at the IndexShard level but they live on their own in RefreshListeners which interacts with IndexShard using a couple of callbacks and a registration method. This lets us test the listeners without standing up an entire IndexShard. We still test the listeners against an InternalEngine, because the interplay between InternalEngine, Translog, and RefreshListeners is complex and important to get right. commitd8926d5fc1Author: Nik Everett <nik9000@gmail.com> Date: Tue May 17 11:02:38 2016 -0400 Move refresh listeners into IndexShard commitdf91cde398Author: Nik Everett <nik9000@gmail.com> Date: Mon May 16 16:01:03 2016 -0400 unused import commit066da45b08Author: Nik Everett <nik9000@gmail.com> Date: Mon May 16 15:54:11 2016 -0400 Remove RefreshListener interface Just pass a Translog.Location and a Consumer<Boolean> when registering. commitb971d6d330Author: Nik Everett <nik9000@gmail.com> Date: Mon May 16 14:41:06 2016 -0400 Docs for setForcedRefresh commit6c43be821eAuthor: Nik Everett <nik9000@gmail.com> Date: Mon May 16 14:34:39 2016 -0400 Rename refresh setter and getter commite61b7391f9Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 22:48:09 2016 -0400 Trigger listeners even when there is no refresh Each refresh gives us an opportunity to pick up any listeners we may have left behind. commit0c9b047708Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 20:30:06 2016 -0400 REST commit8250343240Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 19:34:22 2016 -0400 Switch to estimated count We don't need a linear time count of the number of listeners - a volatile variable is good enough to guess. It probably undercounts more than it overcounts but it isn't a huge problem. commitbd531167feAuthor: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 18:21:02 2016 -0400 Don't try and set forced refresh on bulk items without a response NullPointerExceptions are bad. If the entire request fails then the user has worse problems then "did these force a refresh". commitbcfded1151Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 18:14:20 2016 -0400 Replace LinkedList and synchronized with LinkedTransferQueue commit8a80cc70a7Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 17:38:24 2016 -0400 Support for update commit1f36966742Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 15:46:06 2016 -0400 Cleanup translog tests commit8d121bf35eAuthor: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 15:40:53 2016 -0400 Cleanup listener implementation Much more testing too! commit2058f4a808Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 11:45:55 2016 -0400 Pass back information about whether we refreshed commite445cb0cb9Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 11:03:31 2016 -0400 Javadoc commit611cbeeaebAuthor: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 11:01:40 2016 -0400 Move ReplicationResponse now it is in the same package as its request commit9919758b64Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 11:00:14 2016 -0400 Oh boy that wasn't working commit247cb483c4Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 10:29:37 2016 -0400 Basic block_until_refresh exposed to java client and basic "is it plugged in" style tests. commit46c855c997Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 10:11:10 2016 -0400 Move test to own class commita5ffd892d0Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 07:44:25 2016 -0400 WIP commit213bebb6ecAuthor: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 21:35:52 2016 -0400 Add refresh listeners commita2bc7f30e6Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 21:11:55 2016 -0400 Return last written location from refresh commit85033a8755Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 20:28:21 2016 -0400 Never reply to replica actions while you have the operation lock This last thing was causing periodic test failures because we were replying while we had the operation lock. Now, we probably could get away with that in most cases but the tests don't like it and it isn't a good idea to do network io while you have a lock anyway. So this prevents it. commit1f25cf35e7Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 19:56:18 2016 -0400 Cleanup commit52c5f7c3f0Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 19:33:00 2016 -0400 Add a listener to shard operations commit5b142dc331Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 18:03:52 2016 -0400 Cleanup commit3d22b2d7ceAuthor: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 17:59:55 2016 -0400 Push the listener into shardOperationOnPrimary commit34b378943bAuthor: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 17:48:47 2016 -0400 Doc commitb42b8da968Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 17:45:40 2016 -0400 Don't finish early if the primary finishes early We use a "fake" pending shard that we resolve when the replicas have all started. commit0fc045b56eAuthor: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 17:30:06 2016 -0400 Make performOnPrimary asyncS Instead of returning Tuple<Response, ReplicaRequest> it returns ReplicaRequest and takes a ActionListener<Response> as an argument. We call the listener immediately to preserve backwards compatibility for now. commit80119b9a26Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 16:51:53 2016 -0400 Factor out common code in shardOperationOnPrimary commit0642083676Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 16:32:29 2016 -0400 Factor out common code from shardOperationOnReplica commit8bdc415fedAuthor: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 16:23:28 2016 -0400 Create ReplicatedMutationRequest Superclass for index, delete, and bulkShard requests. commit0f8fa846a2Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 16:10:30 2016 -0400 Create TransportReplicatedMutationAction It is the superclass of replication actions that mutate data: index, delete, and shardBulk. shardFlush and shardRefresh are replication actions but they do not extend TransportReplicatedMutationAction because they don't change the data, only shuffle it around.
48 lines
862 B
Text
48 lines
862 B
Text
[[docs]]
|
|
= Document APIs
|
|
|
|
[partintro]
|
|
--
|
|
|
|
This section describes the following CRUD APIs:
|
|
|
|
.Single document APIs
|
|
* <<docs-index_>>
|
|
* <<docs-get>>
|
|
* <<docs-delete>>
|
|
* <<docs-update>>
|
|
|
|
.Multi-document APIs
|
|
* <<docs-multi-get>>
|
|
* <<docs-bulk>>
|
|
* <<docs-update-by-query>>
|
|
* <<docs-reindex>>
|
|
|
|
NOTE: All CRUD APIs are single-index APIs. The `index` parameter accepts a single
|
|
index name, or an `alias` which points to a single index.
|
|
|
|
--
|
|
|
|
include::docs/index_.asciidoc[]
|
|
|
|
include::docs/get.asciidoc[]
|
|
|
|
include::docs/delete.asciidoc[]
|
|
|
|
include::docs/delete-by-query.asciidoc[]
|
|
|
|
include::docs/update.asciidoc[]
|
|
|
|
include::docs/update-by-query.asciidoc[]
|
|
|
|
include::docs/multi-get.asciidoc[]
|
|
|
|
include::docs/bulk.asciidoc[]
|
|
|
|
include::docs/reindex.asciidoc[]
|
|
|
|
include::docs/termvectors.asciidoc[]
|
|
|
|
include::docs/multi-termvectors.asciidoc[]
|
|
|
|
include::docs/refresh.asciidoc[]
|