pending patch: Re: [HACKERS] HS/SR and smart shutdown
in [PgSql]
|
Prev: [HACKERS] I am interested in the MERGE command implementation as my gSoC project
Next: pending patch: Re: [HACKERS] Streaming replication and pg_xlogfile_name()
From: Fujii Masao on 1 Apr 2010 04:42 On Thu, Apr 1, 2010 at 12:16 AM, Robert Haas <robertmhaas(a)gmail.com> wrote: > On Wed, Mar 31, 2010 at 5:02 AM, Simon Riggs <simon(a)2ndquadrant.com> wrote: >>> > >From what I have seen, the comment about PM_WAIT_BACKENDS is incorrect. >>> > "backends might be waiting for the WAL record that conflicts with their >>> > queries to be replayed". Recovery sometimes waits for backends, but >>> > backends never wait for recovery. >>> >>> Really? As Heikki explained before, backends might wait for the lock >>> taken by the startup process. >>> http://archives.postgresql.org/pgsql-hackers/2010-01/msg02984.php >> >> Backends wait for locks, yes, but they could be waiting for user locks >> also. That is not "waiting for the WAL record", that concept does not >> exist. > > Hmm... this is a good point, on two levels. First, the comment is not > as well-phrased as it could be. Second, I wonder why we can't kill > the startup process and WAL receiver right away, and then wait for the > backends to die off afterwards. I tested whether killing the startup process and walreceiver releases the lock which the backends are waiting for. Unfortunately it doesn't, and the backends have gotten stuck in my box. The behavior which the startup process shuts down without releasing the lock is a bug? BTW, I tested that by compiling postgres with the attached patch and doing the following operations. 1. Make the SR environment 2. Issue some SQLs to the primary psql -h <primary server> =# CREATE TABLE t(i int); =# BEGIN; =# DROP TABLE t; =# SELECT pg_switch_xlog(); (keep this session alive) 3. Issue some SQLs to the standby psql -h <standby server> =# BEGIN; =# SELECT * FROM t; --> waiting 4. Perform smart shutdown on the standby Then the startup process and walreceiver shut down, but the session created in #3 is still waiting. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center
From: Robert Haas on 1 Apr 2010 06:48 On Thu, Apr 1, 2010 at 4:42 AM, Fujii Masao <masao.fujii(a)gmail.com> wrote: > On Thu, Apr 1, 2010 at 12:16 AM, Robert Haas <robertmhaas(a)gmail.com> wrote: >> On Wed, Mar 31, 2010 at 5:02 AM, Simon Riggs <simon(a)2ndquadrant.com> wrote: >>>> > >From what I have seen, the comment about PM_WAIT_BACKENDS is incorrect. >>>> > "backends might be waiting for the WAL record that conflicts with their >>>> > queries to be replayed". Recovery sometimes waits for backends, but >>>> > backends never wait for recovery. >>>> >>>> Really? As Heikki explained before, backends might wait for the lock >>>> taken by the startup process. >>>> http://archives.postgresql.org/pgsql-hackers/2010-01/msg02984.php >>> >>> Backends wait for locks, yes, but they could be waiting for user locks >>> also. That is not "waiting for the WAL record", that concept does not >>> exist. >> >> Hmm... this is a good point, on two levels. First, the comment is not >> as well-phrased as it could be. Second, I wonder why we can't kill >> the startup process and WAL receiver right away, and then wait for the >> backends to die off afterwards. > > I tested whether killing the startup process and walreceiver releases > the lock which the backends are waiting for. Unfortunately it doesn't, > and the backends have gotten stuck in my box. The behavior which the > startup process shuts down without releasing the lock is a bug? I think that what this shows is that the original design of Hot Standby didn't contemplate ever having Hot Standby up without the startup process running. In retrospect, maybe we want to allow that, because a smart shutdown would be more likely to complete in a timely fashion if we stopped replication first and then waited for the backends to die rather than waiting for the backends to die first and then stopping replication. That's because, for so long as replication continues, it may take new locks as well as releasing old ones, to say nothing of using other system resources like CPU and I/O bandwidth. But, for 9.0, I'm not sure we have any real choice, unless making the startup process release locks when it goes away is a very simple change. Assuming that's not the case, I think we should apply this patch with some updates to the comments, document how it works and that it may change in a future release, and add a TODO for 9.1. Thoughts? ....Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Simon Riggs on 1 Apr 2010 07:18 On Thu, 2010-04-01 at 06:48 -0400, Robert Haas wrote: > On Thu, Apr 1, 2010 at 4:42 AM, Fujii Masao <masao.fujii(a)gmail.com> wrote: > > On Thu, Apr 1, 2010 at 12:16 AM, Robert Haas <robertmhaas(a)gmail.com> wrote: > >> On Wed, Mar 31, 2010 at 5:02 AM, Simon Riggs <simon(a)2ndquadrant.com> wrote: > >>>> > >From what I have seen, the comment about PM_WAIT_BACKENDS is incorrect. > >>>> > "backends might be waiting for the WAL record that conflicts with their > >>>> > queries to be replayed". Recovery sometimes waits for backends, but > >>>> > backends never wait for recovery. > >>>> > >>>> Really? As Heikki explained before, backends might wait for the lock > >>>> taken by the startup process. > >>>> http://archives.postgresql.org/pgsql-hackers/2010-01/msg02984.php > >>> > >>> Backends wait for locks, yes, but they could be waiting for user locks > >>> also. That is not "waiting for the WAL record", that concept does not > >>> exist. > >> > >> Hmm... this is a good point, on two levels. First, the comment is not > >> as well-phrased as it could be. Second, I wonder why we can't kill > >> the startup process and WAL receiver right away, and then wait for the > >> backends to die off afterwards. > > > > I tested whether killing the startup process and walreceiver releases > > the lock which the backends are waiting for. Unfortunately it doesn't, > > and the backends have gotten stuck in my box. The behavior which the > > startup process shuts down without releasing the lock is a bug? > > I think that what this shows is that the original design of Hot > Standby didn't contemplate ever having Hot Standby up without the > startup process running. In retrospect, maybe we want to allow that, > because a smart shutdown would be more likely to complete in a timely > fashion if we stopped replication first and then waited for the > backends to die rather than waiting for the backends to die first and > then stopping replication. That's because, for so long as replication > continues, it may take new locks as well as releasing old ones, to say > nothing of using other system resources like CPU and I/O bandwidth. > But, for 9.0, I'm not sure we have any real choice, unless making the > startup process release locks when it goes away is a very simple > change. Assuming that's not the case, I think we should apply this > patch with some updates to the comments, document how it works and > that it may change in a future release, and add a TODO for 9.1. I'm not willing to investigate this further myself at this stage. This looks like risk for little benefit. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Robert Haas on 1 Apr 2010 07:24 On Thu, Apr 1, 2010 at 7:18 AM, Simon Riggs <simon(a)2ndquadrant.com> wrote: > I'm not willing to investigate this further myself at this stage. This > looks like risk for little benefit. That's kind of what I figured. I'll see about fixing up Fujii-san's patch and documenting the behavior; but it won't happen before the weekend because I'm going to be out of town. ....Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Fujii Masao on 13 Apr 2010 09:18
On Thu, Apr 1, 2010 at 8:24 PM, Robert Haas <robertmhaas(a)gmail.com> wrote: > On Thu, Apr 1, 2010 at 7:18 AM, Simon Riggs <simon(a)2ndquadrant.com> wrote: >> I'm not willing to investigate this further myself at this stage. This >> looks like risk for little benefit. > > That's kind of what I figured. I'll see about fixing up Fujii-san's > patch and documenting the behavior; but it won't happen before the > weekend because I'm going to be out of town. I found the bug which makes smart shutdown get stuck for a while: If there is no WAL file available in the standby, walreceiver might be invoked before we have reached the PM_RECOVERY state. We switch to the PM_RECOVERY state after reading the checkpoint record pointed out in the pg_control file. If it's not found, we are in the PM_INIT or PM_START state and start walreceiver to read it from the primary. If smart shutdown is requested at that point, we cannot switch to the PM_WAIT_READONLY state because pmdie() doesn't allow that. So the SIGTERM is never sent to walreceiver, and smart shutdown would get stuck. If the current state is either PM_INIT or PM_START, it's guaranteed that there is no regular backend, so we should kill walreceiver as soon as smart shutdown is requested, I think. The attached patch does that. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center |