[PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION
in [PgSql]
|
Prev: [HACKERS] [PATCH 1/4] Add "COPY ... TO FUNCTION ..." support
Next: Backup history file should be replicated in Streaming Replication?
From: Robert Haas on 23 Nov 2009 17:16 On Mon, Nov 23, 2009 at 4:34 PM, Daniel Farina <dfarina(a)truviso.com> wrote: > Signed-off-by: Daniel Farina <dfarina(a)truviso.com> Thanks for the patch. You may want to take a look at this: http://wiki.postgresql.org/wiki/Submitting_a_Patch I'm fuzzy on what problem this is attempting to solve... as mentioned in the above guidelines, it's usually good to start with some design discussions before writing/submitting code. Also, we prefer that patches be submitted as context diffs and that they not be split up over multiple emails. Thanks, ....Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Daniel Farina on 23 Nov 2009 18:31 On Mon, Nov 23, 2009 at 2:16 PM, Robert Haas <robertmhaas(a)gmail.com> wrote: > On Mon, Nov 23, 2009 at 4:34 PM, Daniel Farina <dfarina(a)truviso.com> wrote: >> Signed-off-by: Daniel Farina <dfarina(a)truviso.com> > > Thanks for the patch. Â You may want to take a look at this: > > http://wiki.postgresql.org/wiki/Submitting_a_Patch > > I'm fuzzy on what problem this is attempting to solve... It seems somewhat strange that the only things COPY can do with its output stream of bytes is exactly two modes that are baked into Postgres in the core. This allows carefully written UDFs to do whatever they will with the stream of bytes, such as sending into a waiting libpq connection. > as mentioned in the above guidelines, it's usually good to start with some design > discussions before writing/submitting code. The patch is derived from functionality in the Truviso postgres-derived database product which is non-optional. This is extruded from that. > Also, we prefer that patches be submitted as context diffs I actually remembered this right after I sent it...sorry about that. > And that they not be split up over multiple emails. With the possible exception of squashing together the test cases into their implementing patches, I would say this is at least two patches. One is to a contrib, the other to core PostgreSQL. It so happens the core addition makes the contrib changes much more obviously useful. fdr -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Daniel Farina on 23 Nov 2009 19:03 On Mon, Nov 23, 2009 at 3:46 PM, Andrew Dunstan <andrew(a)dunslane.net> wrote: > I recently found myself trying to push data through dblink() and ended up > writing code to make a call to the target to call a function which called > back to the source to select the data and insert it. The speedup was > massive, so I'll be interested to dig into the details here. The way the indirection is accomplished here should be very cheap. Overhead should be comparable to the fwrite() call that is used for copying to a file...this is why I mentioned that it would be interesting to make this good enough to be the underlying mechanism of TO STDOUT/TO 'file' to reduce the overall number of mechanisms used to perform COPY TO. > I'm also interested in COPY returning rows as text[], which was discussed > recently. Does this help move us towards that? Yes. Take a look at the tests introduced to core PostgeSQL (see patch 2), where instead of returning a text[] I return just a single text of the verbatim output of the copy. You could imagine making that an SRF instead. It would have to understand COPY row delimiters in whatever mode you were operating in, though. fdr -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Daniel Farina on 23 Nov 2009 19:06 On Mon, Nov 23, 2009 at 4:03 PM, Daniel Farina <dfarina(a)truviso.com> wrote: > Yes. Â Take a look at the tests introduced to core PostgeSQL (see patch > 2), where instead of returning a text[] I return just a single text of > the verbatim output of the copy. Â You could imagine making that an SRF > instead. Â It would have to understand COPY row delimiters in whatever > mode you were operating in, though. Actually, sorry, I lie, but not in a bad way.... Since COPY operates row at a time (rather than a stream of bytes with arbitrary boundaries) you could rely on being passed each record one-at-a-time. You don't have to understand the delimiter. So you could even make a bytea[][] that even contains the binary output, the first dimension being row number, the second being the bytes themselves. The header would pose an interesting problem, though. fdr -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
From: Tom Lane on 23 Nov 2009 19:20
Greg Smith <greg(a)2ndquadrant.com> writes: > Robert Haas wrote: >> I'm fuzzy on what problem this is attempting to solve... as mentioned >> in the above guidelines, it's usually good to start with some design >> discussions before writing/submitting code. > This has been through some heavy design discussions with a few PG > hackers you know and some you don't, they just couldn't release the > result until now. Those discussions don't have a lot of credibility if they didn't take place on the public mailing lists. pgsql-hackers had some preliminary discussions a couple months back on refactoring COPY to allow things like this --- see the thread starting here: http://archives.postgresql.org/pgsql-hackers/2009-09/msg00486.php While I don't think we arrived at any final decisions, I would like to know how this design fits in with what was discussed then. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers |