CVS corruption/mistagging?
in [PgSql]
|
Prev: Bug: Buffer cache is not scan resistant
Next: CREATE DATABASE cannot be executed from a function or multi-command string
From: Stefan Kaltenbrunner on 15 Aug 2007 05:06 Stefan Kaltenbrunner wrote: > Magnus Hagander wrote: >> Tom Lane wrote: >>> Magnus Hagander <magnus(a)hagander.net> writes: >>>> Tom Lane wrote: >>>>> I'd like someone to double-check that though. Also maybe we should back >>>>> up the repository first? >>>> Just for your info: all VMs on tribble, which includes cvs, are backed >>>> up at 02:30 every day, CEST >>> Good, but the salient followup questions to that are (1) backed up to >>> where exactly?, and (2) how many days' past backups could we get at, >>> if we had to? >> 1) The entire VM, with "dump" >> 2) We have today, yesterday and one weekly (copies the daily on monday, >> if I read it right) >> >> They are dumped to a NFS share on this schedule. That NFS share is >> dumped to tape by systems at Conova - I'll let Stefan fill in the >> details about that. > > that's exactly what happens on the backup side here - but maybe I missed > an earlier mail - is something broken on tribble or on the new CVS VM > that requires a restore from backup ? erm -ENOTENOUGHCOFFEE Stefan ---------------------------(end of broadcast)--------------------------- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
From: Peter Eisentraut on 15 Aug 2007 06:11 Am Mittwoch, 15. August 2007 04:20 schrieb Tom Lane: > we should at least log such commands, and maybe disallow to anyone > except Marc's "pgsql" account. I don't think we should disallow it. Or otherwise we might one day be stuck if we need to release while some specific person is on vacation. I never understood why tagging uses a special account anyway. It should be done as the person doing the tagging. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---------------------------(end of broadcast)--------------------------- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
From: Andrew Dunstan on 15 Aug 2007 11:32 Tom Lane wrote: > Magnus Hagander <magnus(a)hagander.net> writes: > >> Tom Lane wrote: >> >>> Good, but the salient followup questions to that are (1) backed up to >>> where exactly?, and (2) how many days' past backups could we get at, >>> if we had to? >>> > > >> They are dumped to a NFS share on this schedule. That NFS share is >> dumped to tape by systems at Conova - I'll let Stefan fill in the >> details about that. >> > > That's good as far as it goes, but seeing that PG is a worldwide > organization now, I wonder whether our primary CVS shouldn't have > backups on several continents. Pardon my paranoia ... but our > collective arses have been saved by offsite backups at least once > already ... > > > Yes, I think we could improve on that. Have we considered more sophisticated solutions that provide incremental backup on a more frequent basis? I'd be inclined to use Bacula or similar (and it uses Postgres for its metadata store :-) ). Ideally I think we'd like to be able fairly easily and quickly to roll the repo (or some portion of it) back to a fairly arbitrary and fairly precise (say within an hour or two) point in recent time. Meanwhile, those of us who rsync the entire repo could easily make rolling backup copies. Arguably this might be better done from a repo made using --numeric-ids. Tarred and compressed it's a shade under 90 Mb, which isn't huge. If people do this at various times of the day we'd get pretty good coverage :-) cheers andrew ---------------------------(end of broadcast)--------------------------- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
From: Stefan Kaltenbrunner on 15 Aug 2007 12:00 Andrew Dunstan wrote: > > > Tom Lane wrote: >> Magnus Hagander <magnus(a)hagander.net> writes: >> >>> Tom Lane wrote: >>> >>>> Good, but the salient followup questions to that are (1) backed up to >>>> where exactly?, and (2) how many days' past backups could we get at, >>>> if we had to? >>>> >> >> >>> They are dumped to a NFS share on this schedule. That NFS share is >>> dumped to tape by systems at Conova - I'll let Stefan fill in the >>> details about that. >>> >> >> That's good as far as it goes, but seeing that PG is a worldwide >> organization now, I wonder whether our primary CVS shouldn't have >> backups on several continents. Pardon my paranoia ... but our >> collective arses have been saved by offsite backups at least once >> already ... >> >> >> > > Yes, I think we could improve on that. Have we considered more > sophisticated solutions that provide incremental backup on a more > frequent basis? I'd be inclined to use Bacula or similar (and it uses > Postgres for its metadata store :-) ). Ideally I think we'd like to be > able fairly easily and quickly to roll the repo (or some portion of it) > back to a fairly arbitrary and fairly precise (say within an hour or > two) point in recent time. well yeah - while I do think that something as complex like bacula is probably overkill for our needs we can certainly improve over the current state. One thing to consider is that we actually have two major scenarios to deal with: 1. simple repo corruption (accident,cvs software bug, admin error) this one might require us to restore the repo from an older backup in the case the corruption cannot be repaired easily. For this we already have myriads of copies of the trees out in the wild but i might be a good idea to keep a number snapshots of the main repo on the CVS-VPS itself (either done every few hours or made as part of the push to anoncvs and svr1). This way we could do a very simple "inplace" recovery on the same running VPS instance with fairly low inpact to all the commiters (and depending parts of teh infrastructure) 2. total loss of the main CVS-VPS (extended power failure, hardware error, OS bug, admin error, fire, some other catastropic event) - in this case we will have to fail over to one of the other project datacenters and for this we need to have full regular copies of the whole VM on external hosts. Stefan ---------------------------(end of broadcast)--------------------------- TIP 6: explain analyze is your friend
From: Stefan Kaltenbrunner on 15 Aug 2007 12:07
Peter Eisentraut wrote: > Am Mittwoch, 15. August 2007 04:20 schrieb Tom Lane: >> we should at least log such commands, and maybe disallow to anyone >> except Marc's "pgsql" account. > > I don't think we should disallow it. Or otherwise we might one day be stuck > if we need to release while some specific person is on vacation. Is this really a problem in practise ? If such a need arises any commiter (or sysadmin) would probably be able to remove that restriction in a few minutes. I think the point here is to prevent such things done by accident(vs. on purpose) which a script like that seems like a fairly simple yet effective solution. Stefan ---------------------------(end of broadcast)--------------------------- TIP 5: don't forget to increase your free space map settings |