About tapes
in [PgSql]
|
Prev: [HACKERS] About tapes
Next: beta3 & the open items list
From: "mac_man2005 on 18 Jun 2010 15:11 Il 18/06/2010 21:00, Robert Haas ha scritto: > On Fri, Jun 18 > Did you read the rest of the comment? It explains how the code avoids this... > > Robert, thanks for your reply. I read the rest of the document, but please take in account that my question wasn't about "avoiding". My question is "in which cases"? I repeat my question. Tuplesort.c and logtape.c DO implement tapes on disk and currently they do not request 2x or 4x of the input space: so, again, in which case implementing tapes on disks requires between 2x and 4x of input space? Thanks for your attention. Manolo. -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Robert Haas on 18 Jun 2010 15:16 On Fri, Jun 18, 2010 at 3:11 PM, mac_man2005(a)hotmail.it <mac_man2005(a)hotmail.it> wrote: > Il 18/06/2010 21:00, Robert Haas ha scritto: >> >> On Fri, Jun 18 >> Did you read the rest of the comment? �It explains how the code avoids >> this... >> >> > > Robert, thanks for your reply. > I read the rest of the document, but please take in account that my question > wasn't about "avoiding". > My question is "in which cases"? > > I repeat my question. Tuplesort.c and logtape.c DO implement tapes on disk > and currently they do not request 2x or 4x of the input space: so, again, in > which case implementing tapes on disks requires between 2x and 4x of input > space? I think that the comment is saying that it *would* take 2x or 4x the input space IF we created a separate file for each input. So instead we don't. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Tom Lane on 18 Jun 2010 15:37 Robert Haas <robertmhaas(a)gmail.com> writes: > On Fri, Jun 18, 2010 at 3:11 PM, mac_man2005(a)hotmail.it > <mac_man2005(a)hotmail.it> wrote: >> I repeat my question. Tuplesort.c and logtape.c DO implement tapes on disk >> and currently they do not request 2x or 4x of the input space: so, again, in >> which case implementing tapes on disks requires between 2x and 4x of input >> space? > I think that the comment is saying that it *would* take 2x or 4x the > input space IF we created a separate file for each input. So instead > we don't. The point of the comment (and indeed of the whole module) is that if we don't want peak space usage to be at least twice the data volume, we have to recycle the space used by "input tapes" before the tapes have been fully read. There's no way to do that if each "tape" is an independent operating-system file. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: "mac_man2005 on 18 Jun 2010 15:46 Ok, so let's try asking the question in another way. Which is the difference between having more than one tape into a file and having one tape per file? I mean, we are peaking runs belonging to different tapes and merge those runs. Moreover, why space is reduced taking in account that we can free the read tuples, when both using one tape per file or more tapes per file? Il 18/06/2010 21:16, Robert Haas ha scritto: > I think that the comment is saying that it *would* take 2x or 4x the > input space IF we created a separate file for each input. So instead > we don't. > > -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Robert Haas on 18 Jun 2010 17:29
On Fri, Jun 18, 2010 at 3:46 PM, mac_man2005(a)hotmail.it <mac_man2005(a)hotmail.it> wrote: > Which is the difference between having more than one tape into a file and > having one tape per file? It makes it easier to recycle space a little at a time. Suppose you're merging two runs of 100 blocks each. You read in a block from each run and write out two output blocks. Now that you've done that, the first block of each of the input runs is garbage and can be recycled - but if the input runs and the output run are in three separate files, there's no easy way to do that. You can truncate a file (and throw away the end) but there's no easy way to throw away the BEGINNING of a file. So you'll probably have to hold on to the entirety of both inputs until you've written the entirety of the output. On the other hand, suppose you have all the blocks in one big file. The first input run is in blocks 1-100; the second is in blocks 101-200. You can read blocks 1 and 101, say, and write the results to blocks 201 and 202, using extra storage, but only a little bit. When you then read blocks 2 and 102, you write the results to blocks 1 and 100, which are no longer needed, because you've already merged them. When you get done with that, blocks 2 and 102 are now no longer needed and can be used to write the next part of the output. Of course, you have to keep track of which order to reread the blocks in when the sort is done: 201, 202, 1, 101, ... but that's a manageable problem. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |