Prev: [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION
Next: Application name patch - v4
From: Heikki Linnakangas on 26 Nov 2009 03:17 Fujii Masao wrote: > On Thu, Nov 26, 2009 at 4:55 PM, Heikki Linnakangas > <heikki.linnakangas(a)enterprisedb.com> wrote: >> Fujii Masao wrote: >>> In current SR, since a backup history file is not replicated, >>> the standby always starts an archive recovery without a backup >>> history file, and a wrong minRecoveryPoint might be used. This >>> is not a problem for SR itself, but would cause trouble when >>> SR cooperates with Hot Standby. >> But the backup history file is included in the base backup you start >> replication from, right? > > No. A backup history file is created by pg_stop_backup(). > So it's not included in the base backup. Ah, I see. Yeah, that needs to be addressed regardless of HS, because you can otherwise start up (= fail over to) the standby too early, before the minimum recovery point has been reached. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Heikki Linnakangas on 18 Dec 2009 11:03 Fujii Masao wrote: > pg_stop_backup deletes the previous backup history file from pg_xlog. > So replication of a backup history file would fail if just one new > online-backup is caused after the base-backup for the standby is taken. > This is too aggressive deletion policy for Streaming Replication, I think. > > So I'd like to change pg_stop_backup so as to delete only backup > history files of four or more generations ago (four is enough?). This is essentially the same problem we have with WAL files and checkpoints. If the standby falls behind too much, without having on open connection to the master all the time, the master will delete old files that are still needed in the standby. I don't think it's worthwhile to modify pg_stop_backup() like that. We should address the general problem. At the moment, you're fine if you also configure WAL archiving and log file shipping, but it would be nice to have some simpler mechanism to avoid the problem. For example, a GUC in master to retain all log files (including backup history files) for X days. Or some way for to register the standby with the master so that the master knows it's out there, and still needs the logs, even when it's not connected. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Heikki Linnakangas on 18 Dec 2009 13:21 Robert Haas wrote: > On Fri, Dec 18, 2009 at 12:22 PM, Florian Pflug <fgp.phlo.org(a)gmail.com> wrote: >> On 18.12.09 17:05 , Robert Haas wrote: >>> On Fri, Dec 18, 2009 at 11:03 AM, Heikki Linnakangas >>> <heikki.linnakangas(a)enterprisedb.com> wrote: >>>> Or some way for to register the standby with the master so that >>>> the master knows it's out there, and still needs the logs, even when >>>> it's not connected. >>> That is the real answer, I think. >> It'd prefer if the slave could automatically fetch a new base backup if it >> falls behind too far to catch up with the available logs. That way, old logs >> don't start piling up on the server if a slave goes offline for a long time. >> >> The slave could for example run a configurable shell script in that case, >> for example. You could then use that to rsync the data directory from the >> server (after a pg_start_backup() of course). > > That would be nice to have too, Yeah, for small databases, it's probably a better tradeoff. The problem with keeping WAL around in the master indefinitely is that you will eventually run out of disk space if the standby disappears for too long. > but it's almost certainly much harder > to implement. In particular, there's no hard and fast rule for > figuring out when you've dropped so far behind that resnapping the > whole thing is faster than replaying the WAL bit by bit. I'd imagine that you take a new base backup only if you have to, ie. the old WAL files the slave needs have already been deleted from the master. > And, of > course, you'll have to take the standby down if you go that route, > whereas trying to catch up the WAL lets it stay up throughout the > process. Good point. > I think (as I did/do with Hot Standby) that the most important thing > here is to get to a point where we have a reasonably good feature that > is of some use, and commit it. It will probably have some annoying > limitations; we can remove those later. I have a feel that what we > have right now is going to be non-robust in the face of network > breaks, but that is a problem that can be fixed by a future patch. Agreed. About a year ago, I was vocal about not relying on the file based shipping, but I don't have a problem with relying on it as an intermediate step, until we add the other options. It's robust as it is, if you set up WAL archiving. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Greg Smith on 19 Dec 2009 12:24 Robert Haas wrote: > I think (as I did/do with Hot Standby) that the most important thing > here is to get to a point where we have a reasonably good feature that > is of some use, and commit it. It will probably have some annoying > limitations; we can remove those later. I have a feel that what we > have right now is going to be non-robust in the face of network > breaks, but that is a problem that can be fixed by a future patch. > Improving robustness in all the situations where you'd like things to be better for replication is a never ending job. As I understand it, a major issue with this patch right now is how it links to the client libpq. That's the sort of problem that can make this uncomittable. As long as the fundamentals are good, it's important not to get lost in optimizing the end UI here if it's at the expense of getting something you can deploy at all in the process. If Streaming Replication ships with a working core but a horribly complicated setup/failover mechanism, that's infinitely better than not shipping at all because resources were diverted toward making things more robust or easier to setup instead. Also, the pool of authors who can work on tweaking the smaller details here is larger than those capable of working on the fundamental streaming replication code. -- Greg Smith 2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support greg(a)2ndQuadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Heikki Linnakangas on 30 Dec 2009 08:26 Fujii Masao wrote: > On Thu, Dec 24, 2009 at 1:39 PM, Fujii Masao <masao.fujii(a)gmail.com> wrote: >> On Wed, Dec 23, 2009 at 7:50 PM, Heikki Linnakangas >> <heikki.linnakangas(a)enterprisedb.com> wrote: >>> Ok. How about writing the history file in pg_stop_backup() for >>> informational purposes only. Ie. never read it, but rely on the WAL >>> records instead. >> Sounds good. I'll make such change as a self-contained patch. > > Done. Please see the attached patch. > > Design: > > * pg_stop_backup writes the backup-end xlog record which contains > the backup starting point. > > * In archive recovery, the startup process doesn't mark the database > as consistent until it has read the backup-end record. > > * A backup history file is still created as in the past, but is never > used. As the patch stands, reachedBackupEnd is never set to true if starting from a restore point after the end-of-backup. We'll need to store the information that we've reached end-of-backup somewhere on disk. Here's is modified patch that adds a new backupStartPoint field to pg_control for that + some other minor editorialization. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
|
Next
|
Last
Pages: 1 2 Prev: [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION Next: Application name patch - v4 |