InvalidXLogRecPtr in docs
in [PgSql]
|
From: Fujii Masao on 10 Jun 2010 03:43 On Thu, Jun 10, 2010 at 4:07 PM, Heikki Linnakangas <heikki.linnakangas(a)enterprisedb.com> wrote: > Ah, I just committed a patch to do the same, before seeing your email. > Thanks anyway. Yeah, thanks a lot! > BTW, the docs claim about pg_last_xlog_location() that "While streaming > replication is in progress this will increase monotonically." That's a bit > misleading: when the replication connection is broken for some reason and we > restart it, we begin streaming from the beginning of the last WAL segment. > So at that moment, pg_last_xlog_location() moves backwards to the beginning > of the WAL segment. > > Should we: > 1. Just document that, > 2. Change pg_last_xlog_location() to not move backwards in that case, or > 3. Change the behavior so that we start streaming at the exact byte location > where we left off? I'm for 2 as follows. diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 26aeca6..f0fd813 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -524,7 +524,8 @@ XLogWalRcvFlush(void) /* Update shared-memory status */ SpinLockAcquire(&walrcv->mutex); - walrcv->receivedUpto = LogstreamResult.Flush; + if (XLByteLT(walrcv->receivedUpto, LogstreamResult.Flush)) + walrcv->receivedUpto = LogstreamResult.Flush; SpinLockRelease(&walrcv->mutex); > I believe that starting from the beginning of the WAL segment is just > paranoia, to avoid creating a WAL file that's missing some data from the > beginning. Right? Only when the recovery starting record (i.e., the record at the checkpoint redo location) is not found, we need to start replication from the beginning of the segment, I think. That is, fetching_ckpt = true case in the following code. > if (PrimaryConnInfo) > { > RequestXLogStreaming( > fetching_ckpt ? RedoStartLSN : *RecPtr, > PrimaryConnInfo); > continue; > } Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Heikki Linnakangas on 10 Jun 2010 04:04 On 10/06/10 10:43, Fujii Masao wrote: > On Thu, Jun 10, 2010 at 4:07 PM, Heikki Linnakangas > <heikki.linnakangas(a)enterprisedb.com> wrote: >> BTW, the docs claim about pg_last_xlog_location() that "While streaming >> replication is in progress this will increase monotonically." That's a bit >> misleading: when the replication connection is broken for some reason and we >> restart it, we begin streaming from the beginning of the last WAL segment. >> So at that moment, pg_last_xlog_location() moves backwards to the beginning >> of the WAL segment. >> >> Should we: >> 1. Just document that, >> 2. Change pg_last_xlog_location() to not move backwards in that case, or >> 3. Change the behavior so that we start streaming at the exact byte location >> where we left off? > > I'm for 2 as follows. > > diff --git a/src/backend/replication/walreceiver.c > b/src/backend/replication/walreceiver.c > index 26aeca6..f0fd813 100644 > --- a/src/backend/replication/walreceiver.c > +++ b/src/backend/replication/walreceiver.c > @@ -524,7 +524,8 @@ XLogWalRcvFlush(void) > > /* Update shared-memory status */ > SpinLockAcquire(&walrcv->mutex); > - walrcv->receivedUpto = LogstreamResult.Flush; > + if (XLByteLT(walrcv->receivedUpto, LogstreamResult.Flush)) > + walrcv->receivedUpto = LogstreamResult.Flush; > SpinLockRelease(&walrcv->mutex); That's not enough, because we set receivedUpto in RequestXlogStreaming() already. >> I believe that starting from the beginning of the WAL segment is just >> paranoia, to avoid creating a WAL file that's missing some data from the >> beginning. Right? > > Only when the recovery starting record (i.e., the record at the checkpoint > redo location) is not found, we need to start replication from the beginning > of the segment, I think. That is, fetching_ckpt = true case in the following > code. > >> if (PrimaryConnInfo) >> { >> RequestXLogStreaming( >> fetching_ckpt ? RedoStartLSN : *RecPtr, >> PrimaryConnInfo); >> continue; >> } Even then, we wouldn't need to start from the beginning of the WAL segment AFAICS. The point is to start from the Redo pointer, not from the checkpoint record, because as soon as we read the checkpoint record we'll need to start applying WAL from the Redo pointer, which is earlier. The WAL file boundaries don't come into play there. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Fujii Masao on 10 Jun 2010 04:37 On Thu, Jun 10, 2010 at 5:04 PM, Heikki Linnakangas <heikki.linnakangas(a)enterprisedb.com> wrote: >>> Should we: >>> 1. Just document that, >>> 2. Change pg_last_xlog_location() to not move backwards in that case, or >>> 3. Change the behavior so that we start streaming at the exact byte >>> location >>> where we left off? >> >> I'm for 2 as follows. >> >> diff --git a/src/backend/replication/walreceiver.c >> b/src/backend/replication/walreceiver.c >> index 26aeca6..f0fd813 100644 >> --- a/src/backend/replication/walreceiver.c >> +++ b/src/backend/replication/walreceiver.c >> @@ -524,7 +524,8 @@ XLogWalRcvFlush(void) >> >> � � � � � � � � /* Update shared-memory status */ >> � � � � � � � � SpinLockAcquire(&walrcv->mutex); >> - � � � � � � � walrcv->receivedUpto = LogstreamResult.Flush; >> + � � � � � � � if (XLByteLT(walrcv->receivedUpto, LogstreamResult.Flush)) >> + � � � � � � � � � � � walrcv->receivedUpto = LogstreamResult.Flush; >> � � � � � � � � SpinLockRelease(&walrcv->mutex); > > That's not enough, because we set receivedUpto in RequestXlogStreaming() > already. Ah, you are right. >>> I believe that starting from the beginning of the WAL segment is just >>> paranoia, to avoid creating a WAL file that's missing some data from the >>> beginning. Right? >> >> Only when the recovery starting record (i.e., the record at the checkpoint >> redo location) is not found, we need to start replication from the >> beginning >> of the segment, I think. That is, fetching_ckpt = true case in the >> following >> code. >> >>> if (PrimaryConnInfo) >>> { >>> � � � �RequestXLogStreaming( >>> � � � � � � � �fetching_ckpt ? RedoStartLSN : *RecPtr, >>> � � � � � � � �PrimaryConnInfo); >>> � � � �continue; >>> } > > Even then, we wouldn't need to start from the beginning of the WAL segment > AFAICS. The point is to start from the Redo pointer, not from the checkpoint > record, because as soon as we read the checkpoint record we'll need to start > applying WAL from the Redo pointer, which is earlier. The WAL file > boundaries don't come into play there. You mean that the WAL file containing the Redo pointer is guaranteed to exist if we could read the checkpoint record, so we don't need to start from the beginning of the segment? This is probably true. But what if we could not read the checkpoint record? In this case, the WAL file containing the Redo pointer also might not exist. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Heikki Linnakangas on 10 Jun 2010 05:10 On 10/06/10 11:37, Fujii Masao wrote: > On Thu, Jun 10, 2010 at 5:04 PM, Heikki Linnakangas > <heikki.linnakangas(a)enterprisedb.com> wrote: >>>> I believe that starting from the beginning of the WAL segment is just >>>> paranoia, to avoid creating a WAL file that's missing some data from the >>>> beginning. Right? >>> >>> Only when the recovery starting record (i.e., the record at the checkpoint >>> redo location) is not found, we need to start replication from the >>> beginning >>> of the segment, I think. That is, fetching_ckpt = true case in the >>> following >>> code. >>> >>>> if (PrimaryConnInfo) >>>> { >>>> RequestXLogStreaming( >>>> fetching_ckpt ? RedoStartLSN : *RecPtr, >>>> PrimaryConnInfo); >>>> continue; >>>> } >> >> Even then, we wouldn't need to start from the beginning of the WAL segment >> AFAICS. The point is to start from the Redo pointer, not from the checkpoint >> record, because as soon as we read the checkpoint record we'll need to start >> applying WAL from the Redo pointer, which is earlier. The WAL file >> boundaries don't come into play there. > > You mean that the WAL file containing the Redo pointer is guaranteed to exist > if we could read the checkpoint record, so we don't need to start from the > beginning of the segment? This is probably true. But what if we could not read > the checkpoint record? In this case, the WAL file containing the Redo pointer > also might not exist. Oh, I think I understand the issue now: we need the header in the beginning of the WAL segment to be valid, even if the first record we're interested in is in the middle of the file. I missed that. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Tom Lane on 10 Jun 2010 10:06 Heikki Linnakangas <heikki.linnakangas(a)enterprisedb.com> writes: > Even then, we wouldn't need to start from the beginning of the WAL > segment AFAICS. The point is to start from the Redo pointer, not from > the checkpoint record, because as soon as we read the checkpoint record > we'll need to start applying WAL from the Redo pointer, which is > earlier. The WAL file boundaries don't come into play there. I don't believe it's a good idea to have SR not write full xlog segment files. Consider for example the following scenario: 1. SR writes some xlog file from the middle. 2. Filesystem says "ah-hah, I know about sparse storage" and doesn't allocate the first half of the file. 3. Failover: slave goes live. 4. xlog file gets recycled for re-use. 5. While reusing the file, we write into the first half ... or try to, but there's no disk space. 6. PANIC. There are probably some other good reasons not to allow incomplete copies of WAL files to exist on the slave system, anyway. I'm not sure if it's worth the trouble, or even a particularly smart idea, to force the output of the status function to be monotonic regardless of what happens underneath. I think removing that claim from the docs altogether is the easiest answer. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
First
|
Prev
|
Next
|
Last
Pages: 1 2 3 Prev: [HACKERS] InvalidXLogRecPtr in docs Next: [HACKERS] fix use of posix_fadvise in xlog.c |