Parallel pg_restore versus old dump files
in [PgSql]
|
From: Greg Stark on 22 Jun 2010 18:52 On Tue, Jun 22, 2010 at 9:07 PM, Tom Lane <tgl(a)sss.pgh.pa.us> wrote: > 3. Perhaps pg_dump ought to emit a warning when it can't seek, instead > of just silently not writing the data offsets. �That behavior was okay > before when lack of data offsets didn't really matter that much, but > lack of data offsets is a serious performance handicap for parallel > restore even after we fix the outright failure condition (because each > worker is going to read through a lot of data to find what it needs). > I'm not terribly familiar with the pg_dump format, but... the usual strategy for storing a TOC on a non-seekable output stream is to store it at the end of the file. So you just accumulate all the offsets in memory as you generate the file and then write the TOC at the end. Of course you need a seekable input stream when you load it then but it would narrow the slow case to when you have a non-seekable output stream when dumping *and* a non-seekable input stream on restore. On the other hand if we didn't notice this dependency when there was only one variable making it depend on two variables would make it that much more obscure when the slow case hits and users wonder why the restore is taking so long. -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Andrew Dunstan on 22 Jun 2010 21:47 Tom Lane wrote: >>> Another possibility is to just remove the inside-the-loop error test >>> altogether: make it just skip till it finds the desired item, and only >>> throw an error if it hits EOF without finding it. In the case that >>> the error test is trying to catch, this would mean significantly more >>> work done before reporting the error, but do we really care? I'm >>> leaning to this solution because it would not require exporting state >>> from the parallel restore control logic. >>> >> Would exporting a bit of state be so bad? >> > > The threaded case seems a bit messy, and frankly I don't believe that > we'd be buying anything. The error case never actually occurs in the real > world, except perhaps on corrupted archive files, so why should we care > about performance for it? > > OK, I can buy that. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Tom Lane on 22 Jun 2010 21:26 Andrew Dunstan <andrew(a)dunslane.net> writes: > Tom Lane wrote: >> In short, parallel pg_restore is guaranteed to fail on any input file >> made with a pre-8.4 pg_dump on Windows. > IIRC, you can reproduce this on Unix too by sending the output of > pg_dump into a pipe. So it's not uniquely a Windows problem. Right. We need to be able to cope, albeit with degraded performance. > As Greg suggests, the solution would be to have a second TOC at the end > of the file with the offsets. Uh, that doesn't fix anything: if you can't seek, a TOC at the end of the file is useless. And the cases where the writer can't seek are likely to be identically the ones where the reader can't seek, viz pg_dump piped to pg_restore (perhaps with some other programs between). >> Another possibility is to just remove the inside-the-loop error test >> altogether: make it just skip till it finds the desired item, and only >> throw an error if it hits EOF without finding it. In the case that >> the error test is trying to catch, this would mean significantly more >> work done before reporting the error, but do we really care? I'm >> leaning to this solution because it would not require exporting state >> from the parallel restore control logic. > Would exporting a bit of state be so bad? The threaded case seems a bit messy, and frankly I don't believe that we'd be buying anything. The error case never actually occurs in the real world, except perhaps on corrupted archive files, so why should we care about performance for it? > For now, yes. But in 9.1 we should write out a second TOC and teach > pg_restore to look for it. I don't think this is useful. >> 4. Is there any value in back-porting the Windows FSEEKO support into >> 8.3 and 8.2? Arguably, not writing the data offsets is a performance >> bug. However a back-port won't do anything for people who are dumping >> with less than the latest minor release of pg_dump, so doing this might >> be largely wasted effort. > I doubt it's worth it, but I could be persuaded otherwise. I'm leaning in that direction too. Anybody who's doing a version upgrade really ought to be using the newer pg_dump version anyway ... regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Andrew Dunstan on 22 Jun 2010 21:02 Tom Lane wrote: > In short, parallel pg_restore is guaranteed to fail on any input file > made with a pre-8.4 pg_dump on Windows. It may be that there's some > other mechanism involved in the reports we've gotten of parallel restore > failing only some of the time, but I'm thinking that the heretofore > unrecognized dependency on pg_dump-time seekability could well explain > those too. > IIRC, you can reproduce this on Unix too by sending the output of pg_dump into a pipe. So it's not uniquely a Windows problem. As Greg suggests, the solution would be to have a second TOC at the end of the file with the offsets. But I think that's way beyond what we should do on the back branches, and really beyond what we should do for 9.0. We should document the limitation. > I see several action items here: > > 1. The error message emitted by _PrintTocData is incredibly misleading. > It needs to be fixed to tell people if the problem is lack of data > offsets rather than lack of seek capability. > Agreed. > Another possibility is to just remove the inside-the-loop error test > altogether: make it just skip till it finds the desired item, and only > throw an error if it hits EOF without finding it. In the case that > the error test is trying to catch, this would mean significantly more > work done before reporting the error, but do we really care? I'm > leaning to this solution because it would not require exporting state > from the parallel restore control logic. > Would exporting a bit of state be so bad? It seems like it would be a bit cleaner, and I'll be surprised if it's terribly difficult. It can be set at the top of parallel_restore(). > 3. Perhaps pg_dump ought to emit a warning when it can't seek, instead > of just silently not writing the data offsets. That behavior was okay > before when lack of data offsets didn't really matter that much, but > lack of data offsets is a serious performance handicap for parallel > restore even after we fix the outright failure condition (because each > worker is going to read through a lot of data to find what it needs). > For now, yes. But in 9.1 we should write out a second TOC and teach pg_restore to look for it. > 4. Is there any value in back-porting the Windows FSEEKO support into > 8.3 and 8.2? Arguably, not writing the data offsets is a performance > bug. However a back-port won't do anything for people who are dumping > with less than the latest minor release of pg_dump, so doing this might > be largely wasted effort. > I doubt it's worth it, but I could be persuaded otherwise. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Magnus Hagander on 23 Jun 2010 05:37 On Wed, Jun 23, 2010 at 03:26, Tom Lane <tgl(a)sss.pgh.pa.us> wrote: > Andrew Dunstan <andrew(a)dunslane.net> writes: >>> 4. Is there any value in back-porting the Windows FSEEKO support into >>> 8.3 and 8.2? �Arguably, not writing the data offsets is a performance >>> bug. �However a back-port won't do anything for people who are dumping >>> with less than the latest minor release of pg_dump, so doing this might >>> be largely wasted effort. > >> I doubt it's worth it, but I could be persuaded otherwise. > > I'm leaning in that direction too. �Anybody who's doing a version > upgrade really ought to be using the newer pg_dump version anyway ... +1 on not backpatching that stuff - it's build system related, so it's kind of fragile on the windows side :-) -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
|
Next
|
Last
Pages: 1 2 Prev: pg_upgrade issues Next: Implementation of Date/Time Input Interpretation |