Listen / Notify - what to do when the queue is full
in [PgSql]
|
From: Robert Haas on 7 Jan 2010 13:40 On Wed, Dec 9, 2009 at 5:43 AM, Joachim Wieland <joe(a)mcknight.de> wrote: > Hi, > > On Mon, Dec 7, 2009 at 5:38 AM, Greg Smith <greg(a)2ndquadrant.com> wrote: >> JI'm going to mark this one "returned with feedback", and I >> do hope that work continues on this patch so it's early in the queue for the >> final CommitFest for this version. It seems like it just needs a bit more >> time to let the design mature and get the kinks worked out and it could turn >> into a useful feature--I know I've wanted NOTIFY with a payload for a years. > > I am perfectly fine with postponing the patch to the next commitfest. To get > some more feedback and to allow everyone to play with it, I am attaching the > latest version of the patch. > > What has changed: > > Transactional processing is now hopefully correct: > > Examples: > > Backend 1: Backend 2: > > transaction starts > NOTIFY foo; > commit starts > transaction starts > LISTEN foo; > commit starts > commit to clog > commit to clog > > => Backend 2 will receive Backend 1's notification. > > > Backend 1: Backend 2: > > transaction starts > NOTIFY foo; > commit starts > transaction starts > UNLISTEN foo; > commit starts > commit to clog > commit to clog > > => Backend 2 will not receive Backend 1's notification. > > This is done by introducing an additional "AsyncCommitOrderLock". It is grabbed > exclusively from transactions that execute LISTEN / UNLISTEN and in shared mode > for transactions that executed NOTIFY only. LISTEN/UNLISTEN transactions then > register the XIDs of the NOTIFYing transactions that are about to commit > at the same time in order to later find out which notifications are visible and > which ones are not. > > If the queue is full, any other transaction that is trying to place a > notification to the queue is rolled back! This is basically a consequence of > the former. There are two warnings that will show up in the log once the queue > is more than 50% full and another one if it is more than 75% full. The biggest > threat to run into a full queue are probably backends that are LISTENing and > are idle in transaction. > > > I have added a function pg_listening() which just contains the names of the > channels that a backend is listening to. > > > I especially invite people who know more about the transactional stuff than I > do to take a close look at what I have done regarding notification visibility. > > > One open question regarding the payload is if we need to limit it to ASCII to > not risk conversion issues between different backend character sets? > > > The second open issue is what we should do regarding 2PC. These options have > been brought up so far: > > 1) allow NOTIFY in 2PC but it can happen that the transaction needs to be > rolled back if the queue is full > 2) disallow NOTIFY in 2PC alltogether > 3) put notifications to the queue on PREPARE TRANSACTION and make backends not > advance their pointers further than those notifications but wait for the > 2PC transaction to commit. 2PC transactions would never fail but you > effectively stop the notification system until the 2PC transaction commits. > > Comments? Joachim - This no longer applies - please rebase, repost, and add a link to the new version to the commitfest app. Jeff - Do you want to continue reviewing this for the next CommitFest, or hand off to another reviewer? Thanks, ....Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Jeff Davis on 7 Jan 2010 14:04 On Thu, 2010-01-07 at 13:40 -0500, Robert Haas wrote: > Joachim - This no longer applies - please rebase, repost, and add a > link to the new version to the commitfest app. > > Jeff - Do you want to continue reviewing this for the next CommitFest, > or hand off to another reviewer? Sure, I'll review it. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Merlin Moncure on 8 Jan 2010 13:19 On Fri, Jan 8, 2010 at 7:48 AM, Joachim Wieland <joe(a)mcknight.de> wrote: > - do we need to limit the payload to pure ASCII ? I think yes, we need > to. I also think we need to reject other payloads with elog(ERROR...). Just noticed this...don't you mean UTF8? Are we going to force non English speaking users to send all payloads in English? merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Stefan Kaltenbrunner on 8 Jan 2010 13:24 Merlin Moncure wrote: > On Fri, Jan 8, 2010 at 7:48 AM, Joachim Wieland <joe(a)mcknight.de> wrote: >> - do we need to limit the payload to pure ASCII ? I think yes, we need >> to. I also think we need to reject other payloads with elog(ERROR...). > > Just noticed this...don't you mean UTF8? Are we going to force non > English speaking users to send all payloads in English? hmm ASCII only sounds weird though doing UTF8 would have obvious conversion problems in some circumstances - what about bytea (or why _do_ we have to limit this to something?). Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Tom Lane on 8 Jan 2010 13:24
Merlin Moncure <mmoncure(a)gmail.com> writes: > On Fri, Jan 8, 2010 at 7:48 AM, Joachim Wieland <joe(a)mcknight.de> wrote: >> - do we need to limit the payload to pure ASCII ? I think yes, we need >> to. I also think we need to reject other payloads with elog(ERROR...). > Just noticed this...don't you mean UTF8? Are we going to force non > English speaking users to send all payloads in English? No, he meant ASCII. Otherwise we're going to have to deal with encoding conversion issues. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |